UDACITY MLND Capstone Project (Tanbir Ahmed)

On Classification of the STL-10 dataset using Deep CNN and Transfer Learning

Content


  1. Modules to import
  2. Device check
  3. Configuration
  4. Load STL-10 Labeled data
  5. Exploratory visualization of the dataset
  6. Categorize labels
  7. Normalize image data
  8. General Model from scratch
  9. Feature extraction and finetuning using pre-trained Imagenet model (ResNet50)
  10. Augmentation for the Model from Scratch
  11. Performance summary of the CNN model from scratch
  12. Conclusion

Required resources

  • Disk space: Approximately 8GB (with models and data saved)
  • Memory: Approximately 16GB
  • CPU/GPU: A GPU is recommended for faster training (we used GTX 1060)
  • Libraries: Keras, Tensorflow, Scikit-learn, OpenCV
  • Data: ./stl10/train_X.bin, ./stl10/train_y.bin, ./stl10/test_X.bin, ./stl10/test_y.bin, ./stl10/unlabeled_X.bin
  • Directories:
    • Bottleneck features (for original train data and test data with ResNet50): ./bottleneck_features
    • Augmented train data (for saving augmented training data): ./augmented_train_data
    • Finetuned model saved (for saving finetuned ResNet50 model): ./best_model_saved
    • Random images from the internet: ./images

1. Modules to import

In [1]:
import numpy as np
from tqdm import tqdm
import os
import shutil
from shutil import copyfile
import scipy
from scipy import misc

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline

import csv
import random
import glob
import cv2

import keras
from keras.utils import np_utils
from keras.callbacks import ModelCheckpoint   
from keras.preprocessing import image
from keras.applications.resnet50 import ResNet50, preprocess_input

from keras.models import Sequential
from keras.layers import Dense, Dropout, Conv2D, GlobalAveragePooling2D, Flatten, BatchNormalization
Using TensorFlow backend.

2. Device check

We use GeForce GTX 1060 GPU for execution of the code. The following code checks availability of available devices.

In [2]:
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 9893526384083690255
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 5026453913
locality {
  bus_id: 1
}
incarnation: 12992959889438061827
physical_device_desc: "device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0"
]

3. Configuration

In [3]:
config = {
    'data_directory' : './stl10',
    'augmented_train_data' : './augmented_train_data',
    'ResNet50' : {
        'name' : 'ResNet50',
        'input_shape' : (197, 197, 3),
        'dataset' : 'STL-10',
        'features_file_train' : './bottleneck_features/ResNet50_STL-10_features_train.npz',
        'features_file_test' : './bottleneck_features/ResNet50_STL-10_features_test.npz',
        'model_file_saved' : './best_model_saved/ResNet50_STL-10_model_best.hdf5'
    }
}

4. Load STL-10 Labeled Data

In [4]:
# load the data
class load_data:    
    def read_labels(path_to_labels):        
        with open(path_to_labels, 'rb') as f:
            labels = np.fromfile(f, dtype=np.uint8) - 1        
        return labels

    def read_all_images(path_to_data):    
        with open(path_to_data, 'rb') as f:
            # read whole file in uint8 chunks
            everything = np.fromfile(f, dtype=np.uint8)
            images = np.reshape(everything, (-1, 3, 96, 96))
            images = np.transpose(images, (0, 3, 2, 1))
        return images

x_train = load_data.read_all_images(config['data_directory'] + '/train_X.bin')    
y_train = load_data.read_labels(config['data_directory'] + '/train_y.bin')    

x_test = load_data.read_all_images(config['data_directory'] + '/test_X.bin')    
y_test = load_data.read_labels(config['data_directory'] + '/test_y.bin')    
        
y_train = np.squeeze(y_train)
print('Data loaded')


print("Number of training examples =", x_train.shape[0])
print("Number of testing examples =", x_test.shape[0])
print("Image data shape =", x_train.shape[1:])
print("Number of classes =", len(set(y_train)))
Data loaded
Number of training examples = 5000
Number of testing examples = 8000
Image data shape = (96, 96, 3)
Number of classes = 10

5. Exploratory visualization of the dataset

In [61]:
class Summary:
    def __init__(self, X_train, y_train, n_classes):
        self.X_train = X_train
        self.y_train = y_train
        self.n_classes = n_classes
        
        self.labels_text = ['Airplane', 'Bird', 'Car', 'Cat', 'Deer', 'Dog', 'Horse', 'Monkey', 'Ship', 'Truck']
        
    def labels_frequency(self):
        # histogram of label frequency
        hist, bins = np.histogram(self.y_train, bins=self.n_classes)
        width = 0.75 * (bins[1] - bins[0])
        center = (bins[:-1] + bins[1:]) / 2
        plt.bar(center, hist, align='center', width=width)
        plt.show()
        
    def random_data(self, n=10):
        images = []
        labels = []
        for i in range(n):
            index = random.randint(0, len(self.X_train))
            image = self.X_train[index]
            images.append(image)
            labels.append(self.y_train[index])
        return images, labels
    
    def images_from_a_folder(self, files):
        images = []
        for i, img in enumerate(glob.glob(files)):
            image = cv2.imread(img)
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
            images.append(image)
        return images
    
    def display_data(self, images, labels, n_rows, n_cols):
        # show image of 'count' random data points
        plt.figure(figsize=(30, 15), dpi=128)
        fig, axs = plt.subplots(n_rows, n_cols, figsize=(30,15))
        fig.subplots_adjust(hspace = .20, wspace=.15)
        axs = axs.ravel()

        for i in range(n_cols*n_rows):
            axs[i].axis('off')
            axs[i].imshow(images[i])
            axs[i].set_title(self.labels_text[labels[i]], fontsize=20)
            
    def image_histograms(self, images, labels, n_rows, n_cols):
        plt.figure(figsize=(30, 15), dpi=128)
        fig, axs = plt.subplots(n_rows, n_cols, figsize=(30,15))
        fig.subplots_adjust(hspace = .20, wspace=.15)
        axs = axs.ravel()

        for i in range(n_cols*n_rows):
            axs[i].axis('off')
            gray_img = cv2.cvtColor(images[i], cv2.COLOR_BGR2GRAY)
            hist = cv2.calcHist([images[i]],[0],None,[96],[0,96])
            axs[i].hist(gray_img.ravel(),96,[0,96])
            axs[i].set_title(self.labels_text[labels[i]], fontsize=20)

5.1. Frequency distribution

In [62]:
# Training data visualization
#########################################################################
s = Summary(x_train, y_train, 10)          

s.labels_frequency()

5.2. Random images

In [7]:
# Display random images with correct labels
images, labels = s.random_data(50)
s.display_data(images, labels, 5, 10)
<Figure size 3840x1920 with 0 Axes>

5.3. Intensity histograms of images in 5.2

In [8]:
s.image_histograms(images, labels, 5, 10)
<Figure size 3840x1920 with 0 Axes>

6. Categorize labels

In [9]:
# one-hot encode the labels
y_train_categorical = np_utils.to_categorical(y_train, 10)
y_test_categorical = np_utils.to_categorical(y_test, 10)
print('Labels processed')
Labels processed

7. Normalize image data

In [10]:
print("Training data mean before normalizing: ", np.mean(x_train))
x_train_normalized = (x_train)/255.0
print("Training data mean after normalizing: ", np.mean(x_train_normalized))

print("Test data mean before normalizing: ", np.mean(x_test))
x_test_normalized = (x_test)/255.0
print("Test data mean after normalizing: ", np.mean(x_test_normalized))
Training data mean before normalizing:  109.919187146
Training data mean after normalizing:  0.431055635865
Test data mean before normalizing:  109.80558034
Test data mean after normalizing:  0.430610118982

8. General Model form Scratch

8.1. Architecture overview

We intend to design a simple architecture of a Convolutional Neural Network (CNN) from the scratch that preserves symmetry with the following properties:

  • Equal number of channels in all the convolutional layers
  • Same Kernel shape for all the convolutional layers
  • Each convolutional layer is followed by a Batch normalization layer and a 2x2 MaxPooling layer
  • Height of the fully connected layer is proportional to the number of output channels in the convolutional layers
  • Each Flattened or fully-connected layer (except the softmax output layer) contains a dropout layer with the same keep-probability

8.2. Function specification

In [11]:
class ModelFromScratch():
    def __init__(self):
        self.model = None
        
    def build(self, input_shape, conv_depths, c_size, fc_heights, act, n_classes):
        from keras.layers.convolutional import Conv2D, MaxPooling2D
        from keras.layers import BatchNormalization, Flatten
        from keras.models import Input, Model, Sequential        

        n_conv = len(conv_depths)
        inp = Input(input_shape) 
        x = inp
        for i in range(n_conv):
            x = Conv2D(conv_depths[i], (c_size,c_size), activation=act)(x)
            x = BatchNormalization()(x)
            x = MaxPooling2D((2,2), strides=(2, 2))(x)

        x = Flatten()(x)
        x = Dropout(0.5)(x)

        n_fc = len(fc_heights)
        for i in range(n_fc):
            x = Dense(fc_heights[i], activation=act)(x)
            x = Dropout(0.5)(x)

        x = Dense(n_classes, activation='softmax')(x)

        self.model = Model(inputs=inp, outputs=x)
    
    def compile(self, rate, beta_1, beta_2):
        opt = keras.optimizers.Adam(lr=rate, beta_1=beta_1, beta_2=beta_2)
        self.model.compile(loss='categorical_crossentropy', 
                           optimizer=opt, 
                           metrics=['accuracy'])
        
    def fit(self, file, epochs):
        checkpointer = ModelCheckpoint(filepath=file, monitor='val_acc', verbose=1, save_best_only=True)

        import os
        if os.path.isfile(file):
            self.model.load_weights(file)
        self.model.fit(x_train_normalized, y_train_categorical, batch_size=128, epochs=epochs, validation_split=0.2, 
                       callbacks=[checkpointer], verbose=2, shuffle=True)  

    def fit_augmented(self, file, epochs):
        checkpointer = ModelCheckpoint(filepath=file, monitor='val_acc', verbose=1, save_best_only=True)

        import os
        if os.path.isfile(file):
            self.model.load_weights(file)
        self.model.fit(x_train_augmented_normalized, y_train_augmented_categorical, 
                       batch_size=128, epochs=epochs, validation_split=0.2, 
                       callbacks=[checkpointer], verbose=2, shuffle=True)     
        
    def test(self, file):    
        self.model.load_weights(file)
        # evaluate test accuracy
        score = self.model.evaluate(x_test_normalized, y_test_categorical, verbose=2)
        accuracy = 100*score[1]

        # print test accuracy
        print('Test accuracy: %.4f%%' % accuracy)
        
    def predict(self, img):
        pred = self.model.predict(img)
        return pred
        
    def summary(self):
        self.model.summary()

8.3. Build, Compile, Fit, and Test

Build and Compile

Define the following variables:

  • $N$ = number of convolutional layers
  • $m$ = number of channels in each of the $N$ convolutional layers
  • $t$ for the kernel shape ($t\times t$)
  • $m\times 16$ = size of the only fully-connected layer before the output layer

Upon tuning over the following values

  • $N \in \{3,4\}$,
  • $m \in \{32, 48, 64, 80, 96\}$, and
  • $t \in \{3,4,5\}$,

we have found consistantly above 60% test accuracy with $N=3$, $m=80$, and $t=4$.

In [12]:
model = ModelFromScratch()
model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
model.summary()
model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 96, 96, 3)         0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 93, 93, 80)        3920      
_________________________________________________________________
batch_normalization_1 (Batch (None, 93, 93, 80)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 46, 46, 80)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 43, 43, 80)        102480    
_________________________________________________________________
batch_normalization_2 (Batch (None, 43, 43, 80)        320       
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 21, 21, 80)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 18, 18, 80)        102480    
_________________________________________________________________
batch_normalization_3 (Batch (None, 18, 18, 80)        320       
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 9, 9, 80)          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 6480)              0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 6480)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 1280)              8295680   
_________________________________________________________________
dropout_2 (Dropout)          (None, 1280)              0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                12810     
=================================================================
Total params: 8,518,330
Trainable params: 8,517,850
Non-trainable params: 480
_________________________________________________________________

Fit and Test

8.3.1. Fit original training subset (5000 x 96 x 96 x 3)

In [33]:
if os.path.isfile('model_80x3_4_1280_relu_original_5000.hdf5'):
    pass
else:
    model.fit('model_80x3_4_1280_relu_original_5000.hdf5', epochs=50)
Train on 4000 samples, validate on 1000 samples
Epoch 1/50
 - 11s - loss: 3.9772 - acc: 0.2472 - val_loss: 2.0534 - val_acc: 0.3950

Epoch 00001: val_acc improved from -inf to 0.39500, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 2/50
 - 10s - loss: 2.6860 - acc: 0.3623 - val_loss: 1.6759 - val_acc: 0.4540

Epoch 00002: val_acc improved from 0.39500 to 0.45400, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 3/50
 - 10s - loss: 2.1246 - acc: 0.4210 - val_loss: 1.5248 - val_acc: 0.4520

Epoch 00003: val_acc did not improve
Epoch 4/50
 - 10s - loss: 1.8530 - acc: 0.4517 - val_loss: 1.4379 - val_acc: 0.4640

Epoch 00004: val_acc improved from 0.45400 to 0.46400, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 5/50
 - 10s - loss: 1.6386 - acc: 0.4903 - val_loss: 1.3738 - val_acc: 0.4950

Epoch 00005: val_acc improved from 0.46400 to 0.49500, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 6/50
 - 10s - loss: 1.4060 - acc: 0.5230 - val_loss: 1.3591 - val_acc: 0.5020

Epoch 00006: val_acc improved from 0.49500 to 0.50200, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 7/50
 - 10s - loss: 1.3220 - acc: 0.5550 - val_loss: 1.3158 - val_acc: 0.5150

Epoch 00007: val_acc improved from 0.50200 to 0.51500, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 8/50
 - 10s - loss: 1.2120 - acc: 0.5780 - val_loss: 1.2878 - val_acc: 0.5340

Epoch 00008: val_acc improved from 0.51500 to 0.53400, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 9/50
 - 10s - loss: 1.1301 - acc: 0.6028 - val_loss: 1.2826 - val_acc: 0.5430

Epoch 00009: val_acc improved from 0.53400 to 0.54300, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 10/50
 - 10s - loss: 1.0733 - acc: 0.6178 - val_loss: 1.2544 - val_acc: 0.5650

Epoch 00010: val_acc improved from 0.54300 to 0.56500, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 11/50
 - 10s - loss: 0.9913 - acc: 0.6445 - val_loss: 1.2523 - val_acc: 0.5590

Epoch 00011: val_acc did not improve
Epoch 12/50
 - 10s - loss: 0.9496 - acc: 0.6615 - val_loss: 1.2063 - val_acc: 0.5640

Epoch 00012: val_acc did not improve
Epoch 13/50
 - 10s - loss: 0.9069 - acc: 0.6787 - val_loss: 1.1994 - val_acc: 0.5630

Epoch 00013: val_acc did not improve
Epoch 14/50
 - 10s - loss: 0.8036 - acc: 0.7120 - val_loss: 1.2062 - val_acc: 0.5620

Epoch 00014: val_acc did not improve
Epoch 15/50
 - 10s - loss: 0.7731 - acc: 0.7230 - val_loss: 1.1751 - val_acc: 0.5760

Epoch 00015: val_acc improved from 0.56500 to 0.57600, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 16/50
 - 10s - loss: 0.7288 - acc: 0.7392 - val_loss: 1.1656 - val_acc: 0.5760

Epoch 00016: val_acc improved from 0.57600 to 0.57600, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 17/50
 - 10s - loss: 0.6666 - acc: 0.7678 - val_loss: 1.1573 - val_acc: 0.5990

Epoch 00017: val_acc improved from 0.57600 to 0.59900, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 18/50
 - 10s - loss: 0.6392 - acc: 0.7762 - val_loss: 1.1555 - val_acc: 0.6030

Epoch 00018: val_acc improved from 0.59900 to 0.60300, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 19/50
 - 10s - loss: 0.6285 - acc: 0.7770 - val_loss: 1.1593 - val_acc: 0.5920

Epoch 00019: val_acc did not improve
Epoch 20/50
 - 10s - loss: 0.5735 - acc: 0.7998 - val_loss: 1.1417 - val_acc: 0.6120

Epoch 00020: val_acc improved from 0.60300 to 0.61200, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 21/50
 - 10s - loss: 0.5571 - acc: 0.7923 - val_loss: 1.1267 - val_acc: 0.6090

Epoch 00021: val_acc did not improve
Epoch 22/50
 - 10s - loss: 0.5173 - acc: 0.8220 - val_loss: 1.1380 - val_acc: 0.6120

Epoch 00022: val_acc did not improve
Epoch 23/50
 - 10s - loss: 0.4734 - acc: 0.8347 - val_loss: 1.1250 - val_acc: 0.6110

Epoch 00023: val_acc did not improve
Epoch 24/50
 - 10s - loss: 0.4502 - acc: 0.8393 - val_loss: 1.1308 - val_acc: 0.6050

Epoch 00024: val_acc did not improve
Epoch 25/50
 - 10s - loss: 0.4115 - acc: 0.8580 - val_loss: 1.1505 - val_acc: 0.5960

Epoch 00025: val_acc did not improve
Epoch 26/50
 - 10s - loss: 0.4006 - acc: 0.8530 - val_loss: 1.1117 - val_acc: 0.6150

Epoch 00026: val_acc improved from 0.61200 to 0.61500, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 27/50
 - 10s - loss: 0.3625 - acc: 0.8688 - val_loss: 1.1445 - val_acc: 0.6130

Epoch 00027: val_acc did not improve
Epoch 28/50
 - 10s - loss: 0.3412 - acc: 0.8780 - val_loss: 1.1578 - val_acc: 0.6130

Epoch 00028: val_acc did not improve
Epoch 29/50
 - 10s - loss: 0.3303 - acc: 0.8900 - val_loss: 1.1310 - val_acc: 0.6240

Epoch 00029: val_acc improved from 0.61500 to 0.62400, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 30/50
 - 10s - loss: 0.3175 - acc: 0.8898 - val_loss: 1.1347 - val_acc: 0.6130

Epoch 00030: val_acc did not improve
Epoch 31/50
 - 10s - loss: 0.2977 - acc: 0.8985 - val_loss: 1.1756 - val_acc: 0.6160

Epoch 00031: val_acc did not improve
Epoch 32/50
 - 10s - loss: 0.2747 - acc: 0.9035 - val_loss: 1.1812 - val_acc: 0.6150

Epoch 00032: val_acc did not improve
Epoch 33/50
 - 10s - loss: 0.2610 - acc: 0.9158 - val_loss: 1.2082 - val_acc: 0.6080

Epoch 00033: val_acc did not improve
Epoch 34/50
 - 10s - loss: 0.2544 - acc: 0.9100 - val_loss: 1.1606 - val_acc: 0.6260

Epoch 00034: val_acc improved from 0.62400 to 0.62600, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 35/50
 - 10s - loss: 0.2247 - acc: 0.9265 - val_loss: 1.1795 - val_acc: 0.6190

Epoch 00035: val_acc did not improve
Epoch 36/50
 - 10s - loss: 0.2189 - acc: 0.9267 - val_loss: 1.1986 - val_acc: 0.6180

Epoch 00036: val_acc did not improve
Epoch 37/50
 - 10s - loss: 0.2127 - acc: 0.9240 - val_loss: 1.1970 - val_acc: 0.6210

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 10s - loss: 0.2074 - acc: 0.9300 - val_loss: 1.1934 - val_acc: 0.6250

Epoch 00038: val_acc did not improve
Epoch 39/50
 - 10s - loss: 0.1822 - acc: 0.9405 - val_loss: 1.1914 - val_acc: 0.6230

Epoch 00039: val_acc did not improve
Epoch 40/50
 - 10s - loss: 0.1719 - acc: 0.9442 - val_loss: 1.2027 - val_acc: 0.6100

Epoch 00040: val_acc did not improve
Epoch 41/50
 - 10s - loss: 0.1622 - acc: 0.9455 - val_loss: 1.2212 - val_acc: 0.6120

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 10s - loss: 0.1580 - acc: 0.9480 - val_loss: 1.2208 - val_acc: 0.6280

Epoch 00042: val_acc improved from 0.62600 to 0.62800, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 43/50
 - 10s - loss: 0.1499 - acc: 0.9523 - val_loss: 1.2353 - val_acc: 0.6220

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 10s - loss: 0.1319 - acc: 0.9592 - val_loss: 1.2711 - val_acc: 0.6140

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 10s - loss: 0.1304 - acc: 0.9562 - val_loss: 1.1933 - val_acc: 0.6230

Epoch 00045: val_acc did not improve
Epoch 46/50
 - 10s - loss: 0.1259 - acc: 0.9610 - val_loss: 1.1916 - val_acc: 0.6330

Epoch 00046: val_acc improved from 0.62800 to 0.63300, saving model to model_80x3_4_1280_relu_original_5000.hdf5
Epoch 47/50
 - 10s - loss: 0.1225 - acc: 0.9625 - val_loss: 1.2374 - val_acc: 0.6130

Epoch 00047: val_acc did not improve
Epoch 48/50
 - 10s - loss: 0.1219 - acc: 0.9610 - val_loss: 1.2348 - val_acc: 0.6170

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 10s - loss: 0.1177 - acc: 0.9597 - val_loss: 1.2436 - val_acc: 0.6160

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 10s - loss: 0.1113 - acc: 0.9645 - val_loss: 1.2523 - val_acc: 0.6200

Epoch 00050: val_acc did not improve

8.3.2. Test using weights trained from original data (5000 x 96 x96 x3)

In [34]:
if os.path.isfile('model_80x3_4_1280_relu_original_5000.hdf5'):
    model.test('model_80x3_4_1280_relu_original_5000.hdf5')
else:
    pass
Test accuracy: 61.6000%

Comment

We have been able to obtain a test accuracy more than 60% on the test subset with the weights trained on the original training dataset which was part of the goal of the project.

9. Feature extraction and finetuning using a pre-trained Imagenet model

We use the ResNet50 pre-trained Imagenet model to extract features from STL-10 training dataset and then finetune it to use it as a classifier of the STL-10 dataset. We use the classifier (with at least 90% accuracy on the STL-10 test subset) to label the STL-10 unlabeled dataset.

In [35]:
class FeaturesExtractor:
    def __init__(self, 
                 model_config):
        self.model_config = model_config
        self.name = model_config['name']
        self.input_shape = model_config['input_shape']        
        self.features_file_train = model_config['features_file_train']
        self.features_file_test = model_config['features_file_test']
        self.model_file_saved = model_config['model_file_saved']
        
        self.model = ResNet50(weights='imagenet', include_top=False, input_shape=self.input_shape)            
        print('Output shape:', self.model.output_shape)
        print('model loaded')

    
    def extract_imageset_features(self, images, file=None):
        print('Reshaping images')
        resized_images = np.array([scipy.misc.imresize(images[i], self.input_shape) 
                            for i in tqdm(range(0, len(images)))]).astype('float32')
        print('Images resized')
        
        print('Preprocessing images')
        preprocessed_input = preprocess_input(resized_images)        
        print('Images preprocessed')
        
        print('Features extraction - starts')
        features = []
        for i in tqdm(range(0, len(images))):            
            img_features = self.model.predict(np.expand_dims(resized_images[i], axis=0))         
            features.append(img_features)
        features = np.squeeze(features)         
        print('Features extraction - completed')
        
        if file != None:
            print('Saving features...')
            np.savez(file, features=features)
            print('Features saved')
        return features
    
    def extract_image_features(self, image, file=None):
        resized_image = scipy.misc.imresize(image, self.input_shape).astype('float32')
        preprocessed_input = preprocess_input(resized_image)        
        features = self.model.predict(np.expand_dims(resized_image, axis=0))         
        features = np.squeeze(features)         
        return features
    
    def extract_train_features(self, X):
        if os.path.exists(self.features_file_train):
            print('Bottleneck features detected (train)')
            features = np.load(self.features_file_train)['features']
            print('Bottleneck features (train) loaded')
        else:
            features = self.extract_imageset_features(X, file=self.features_file_train)
        return features
            
    def extract_test_features(self, X):
        if os.path.exists(self.features_file_test):
            print('Bottleneck features detected (test)')
            features = np.load(self.features_file_test)['features']
            print('Bottleneck features (test) loaded')
        else:
            features = self.extract_imageset_features(X, file=self.features_file_test)
        return features

    
    def extract_unlabeled_imageset_features(self, images):
        resized_images = np.array([scipy.misc.imresize(images[i], self.input_shape) 
                            for i in range(0, len(images))]).astype('float32')
        preprocessed_input = preprocess_input(resized_images)        
        
        print('Features extraction - starts')
        features = []
        for i in tqdm(range(0, len(images))):            
            img_features = self.model.predict(np.expand_dims(resized_images[i], axis=0))         
            features.append(img_features)
        features = np.squeeze(features)         
        print('Features extraction - completed')
        return features
    
    def extract_unlabeled_features(self, X, i):
        file = self.model_config["features_file_unlabeled_"+i]
        if os.path.exists(file):
            print('Bottleneck features detected (unlabeled)')
            features = np.load(file)['features']
            print('Bottleneck features (unlabeled) loaded')
        else:
            features = self.extract_imageset_features(X, file=file)
        return features

9.1. Feature Extraction

In [36]:
# ResNet50 has output shape = 1x1x2048 
x = FeaturesExtractor(config['ResNet50'])
Output shape: (None, 1, 1, 2048)
model loaded

9.2. Training features

In [37]:
train_features = x.extract_train_features(x_train)
Bottleneck features detected (train)
Bottleneck features (train) loaded

9.2.1. TSNE embedding of bottleneck features for Training set

In [38]:
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
%matplotlib inline

# reshape bottleneck features + reduce dimensionality with t-SNE
if os.path.exists('tsne_features.npz'):
    print('tsne features detected (test)')
    tsne_features = np.load('tsne_features.npz')['tsne_features']
else:
    print('tsne features not detected (test)')
    print('calculating now ...') 
    # fit_trainform: transform data into lower dimension
    tsne_features = TSNE(learning_rate=100).fit_transform(train_features) 
    np.savez('tsne_features', tsne_features=tsne_features)
print('tsne features obtained')

# plot the features
plt.figure(figsize=(12,12))
plt.scatter(tsne_features[:,0], tsne_features[:,1], c=plt.cm.jet(y_train/10), s=10, edgecolors='none')
plt.show()
tsne features detected (test)
tsne features obtained

Analysis:

The bottleneck features of ResNet50 provides encouraging result as shown in the plot above where points representing same-class objects are in the neighborhood on the 2D plot. Thus training a classifier on the bottleneck features would provide better validation and test performance.

9.2.2. Finetuning and classification layers with bottleneck features as input

In [39]:
def Model(input_shape):
    model = Sequential()    
    model.add(Dense(512, activation='relu', input_shape=input_shape))
    model.add(BatchNormalization(axis=-1))
    model.add(Dropout(0.5))
    
    # 6 gives 93.850% test acc
    for i in range(6):
        model.add(Dense(256, activation='relu'))
        model.add(BatchNormalization(axis=-1))
        #model.add(Dropout(0.3))
    
    model.add(Dense(10, activation='softmax'))
    return model

9.2.3. Compile

In [40]:
modelResNet50 = Model(train_features.shape[1:])
modelResNet50.summary()

modelResNet50.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 512)               1049088   
_________________________________________________________________
batch_normalization_1 (Batch (None, 512)               2048      
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 256)               131328    
_________________________________________________________________
batch_normalization_2 (Batch (None, 256)               1024      
_________________________________________________________________
dense_3 (Dense)              (None, 256)               65792     
_________________________________________________________________
batch_normalization_3 (Batch (None, 256)               1024      
_________________________________________________________________
dense_4 (Dense)              (None, 256)               65792     
_________________________________________________________________
batch_normalization_4 (Batch (None, 256)               1024      
_________________________________________________________________
dense_5 (Dense)              (None, 256)               65792     
_________________________________________________________________
batch_normalization_5 (Batch (None, 256)               1024      
_________________________________________________________________
dense_6 (Dense)              (None, 256)               65792     
_________________________________________________________________
batch_normalization_6 (Batch (None, 256)               1024      
_________________________________________________________________
dense_7 (Dense)              (None, 256)               65792     
_________________________________________________________________
batch_normalization_7 (Batch (None, 256)               1024      
_________________________________________________________________
dense_8 (Dense)              (None, 10)                2570      
=================================================================
Total params: 1,520,138
Trainable params: 1,516,042
Non-trainable params: 4,096
_________________________________________________________________

9.2.4. Fit

In [206]:
checkpointer = ModelCheckpoint(filepath=config['ResNet50']['model_file_saved'], monitor='val_acc', 
                               verbose=1, save_best_only=True)

if os.path.isfile(config['ResNet50']['model_file_saved']):
    modelResNet50.load_weights(config['ResNet50']['model_file_saved'])
    
modelResNet50.fit(train_features, y_train_categorical, batch_size=30, epochs=100, validation_split=0.2, 
                  callbacks=[checkpointer], verbose=2, shuffle=True)
Train on 4000 samples, validate on 1000 samples
Epoch 1/100
 - 3s - loss: 0.0292 - acc: 0.9913 - val_loss: 0.3221 - val_acc: 0.9100

Epoch 00001: val_acc improved from -inf to 0.91000, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 2/100
 - 3s - loss: 0.0181 - acc: 0.9945 - val_loss: 0.2955 - val_acc: 0.9280

Epoch 00002: val_acc improved from 0.91000 to 0.92800, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 3/100
 - 3s - loss: 0.0170 - acc: 0.9943 - val_loss: 0.3319 - val_acc: 0.9240

Epoch 00003: val_acc did not improve
Epoch 4/100
 - 3s - loss: 0.0252 - acc: 0.9933 - val_loss: 0.4232 - val_acc: 0.9150

Epoch 00004: val_acc did not improve
Epoch 5/100
 - 3s - loss: 0.0278 - acc: 0.9925 - val_loss: 0.3585 - val_acc: 0.9240

Epoch 00005: val_acc did not improve
Epoch 6/100
 - 3s - loss: 0.0188 - acc: 0.9940 - val_loss: 0.3831 - val_acc: 0.9180

Epoch 00006: val_acc did not improve
Epoch 7/100
 - 3s - loss: 0.0183 - acc: 0.9950 - val_loss: 0.3547 - val_acc: 0.9240

Epoch 00007: val_acc did not improve
Epoch 8/100
 - 3s - loss: 0.0083 - acc: 0.9983 - val_loss: 0.3210 - val_acc: 0.9280

Epoch 00008: val_acc did not improve
Epoch 9/100
 - 3s - loss: 0.0103 - acc: 0.9973 - val_loss: 0.3113 - val_acc: 0.9260

Epoch 00009: val_acc did not improve
Epoch 10/100
 - 3s - loss: 0.0278 - acc: 0.9923 - val_loss: 0.3092 - val_acc: 0.9250

Epoch 00010: val_acc did not improve
Epoch 11/100
 - 3s - loss: 0.0100 - acc: 0.9963 - val_loss: 0.3110 - val_acc: 0.9240

Epoch 00011: val_acc did not improve
Epoch 12/100
 - 3s - loss: 0.0066 - acc: 0.9983 - val_loss: 0.3192 - val_acc: 0.9230

Epoch 00012: val_acc did not improve
Epoch 13/100
 - 3s - loss: 0.0121 - acc: 0.9960 - val_loss: 0.3305 - val_acc: 0.9250

Epoch 00013: val_acc did not improve
Epoch 14/100
 - 3s - loss: 0.0098 - acc: 0.9968 - val_loss: 0.3270 - val_acc: 0.9200

Epoch 00014: val_acc did not improve
Epoch 15/100
 - 3s - loss: 0.0181 - acc: 0.9948 - val_loss: 0.3399 - val_acc: 0.9240

Epoch 00015: val_acc did not improve
Epoch 16/100
 - 3s - loss: 0.0234 - acc: 0.9930 - val_loss: 0.3556 - val_acc: 0.9250

Epoch 00016: val_acc did not improve
Epoch 17/100
 - 3s - loss: 0.0062 - acc: 0.9978 - val_loss: 0.3685 - val_acc: 0.9260

Epoch 00017: val_acc did not improve
Epoch 18/100
 - 3s - loss: 0.0136 - acc: 0.9958 - val_loss: 0.3178 - val_acc: 0.9250

Epoch 00018: val_acc did not improve
Epoch 19/100
 - 3s - loss: 0.0146 - acc: 0.9963 - val_loss: 0.3694 - val_acc: 0.9210

Epoch 00019: val_acc did not improve
Epoch 20/100
 - 3s - loss: 0.0152 - acc: 0.9960 - val_loss: 0.3524 - val_acc: 0.9160

Epoch 00020: val_acc did not improve
Epoch 21/100
 - 3s - loss: 0.0104 - acc: 0.9973 - val_loss: 0.3838 - val_acc: 0.9140

Epoch 00021: val_acc did not improve
Epoch 22/100
 - 3s - loss: 0.0107 - acc: 0.9968 - val_loss: 0.3826 - val_acc: 0.9150

Epoch 00022: val_acc did not improve
Epoch 23/100
 - 3s - loss: 0.0114 - acc: 0.9963 - val_loss: 0.3121 - val_acc: 0.9320

Epoch 00023: val_acc improved from 0.92800 to 0.93200, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 24/100
 - 3s - loss: 0.0117 - acc: 0.9970 - val_loss: 0.3444 - val_acc: 0.9210

Epoch 00024: val_acc did not improve
Epoch 25/100
 - 3s - loss: 0.0088 - acc: 0.9965 - val_loss: 0.3731 - val_acc: 0.9210

Epoch 00025: val_acc did not improve
Epoch 26/100
 - 3s - loss: 0.0055 - acc: 0.9985 - val_loss: 0.3702 - val_acc: 0.9200

Epoch 00026: val_acc did not improve
Epoch 27/100
 - 3s - loss: 0.0140 - acc: 0.9955 - val_loss: 0.3726 - val_acc: 0.9310

Epoch 00027: val_acc did not improve
Epoch 28/100
 - 3s - loss: 0.0126 - acc: 0.9955 - val_loss: 0.4241 - val_acc: 0.9150

Epoch 00028: val_acc did not improve
Epoch 29/100
 - 3s - loss: 0.0107 - acc: 0.9970 - val_loss: 0.3311 - val_acc: 0.9310

Epoch 00029: val_acc did not improve
Epoch 30/100
 - 3s - loss: 0.0126 - acc: 0.9973 - val_loss: 0.3375 - val_acc: 0.9340

Epoch 00030: val_acc improved from 0.93200 to 0.93400, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 31/100
 - 3s - loss: 0.0138 - acc: 0.9958 - val_loss: 0.3677 - val_acc: 0.9350

Epoch 00031: val_acc improved from 0.93400 to 0.93500, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 32/100
 - 3s - loss: 0.0142 - acc: 0.9955 - val_loss: 0.3344 - val_acc: 0.9210

Epoch 00032: val_acc did not improve
Epoch 33/100
 - 3s - loss: 0.0137 - acc: 0.9960 - val_loss: 0.3363 - val_acc: 0.9260

Epoch 00033: val_acc did not improve
Epoch 34/100
 - 3s - loss: 0.0062 - acc: 0.9978 - val_loss: 0.3352 - val_acc: 0.9190

Epoch 00034: val_acc did not improve
Epoch 35/100
 - 3s - loss: 0.0098 - acc: 0.9968 - val_loss: 0.3253 - val_acc: 0.9220

Epoch 00035: val_acc did not improve
Epoch 36/100
 - 3s - loss: 0.0202 - acc: 0.9950 - val_loss: 0.3224 - val_acc: 0.9290

Epoch 00036: val_acc did not improve
Epoch 37/100
 - 3s - loss: 0.0217 - acc: 0.9933 - val_loss: 0.3468 - val_acc: 0.9220

Epoch 00037: val_acc did not improve
Epoch 38/100
 - 3s - loss: 0.0097 - acc: 0.9968 - val_loss: 0.3363 - val_acc: 0.9200

Epoch 00038: val_acc did not improve
Epoch 39/100
 - 3s - loss: 0.0115 - acc: 0.9965 - val_loss: 0.3060 - val_acc: 0.9350

Epoch 00039: val_acc did not improve
Epoch 40/100
 - 3s - loss: 0.0125 - acc: 0.9970 - val_loss: 0.3038 - val_acc: 0.9310

Epoch 00040: val_acc did not improve
Epoch 41/100
 - 3s - loss: 0.0061 - acc: 0.9988 - val_loss: 0.3147 - val_acc: 0.9320

Epoch 00041: val_acc did not improve
Epoch 42/100
 - 3s - loss: 0.0057 - acc: 0.9978 - val_loss: 0.3092 - val_acc: 0.9330

Epoch 00042: val_acc did not improve
Epoch 43/100
 - 3s - loss: 0.0096 - acc: 0.9965 - val_loss: 0.3367 - val_acc: 0.9310

Epoch 00043: val_acc did not improve
Epoch 44/100
 - 3s - loss: 0.0088 - acc: 0.9970 - val_loss: 0.3133 - val_acc: 0.9240

Epoch 00044: val_acc did not improve
Epoch 45/100
 - 3s - loss: 0.0146 - acc: 0.9953 - val_loss: 0.3447 - val_acc: 0.9280

Epoch 00045: val_acc did not improve
Epoch 46/100
 - 3s - loss: 0.0131 - acc: 0.9963 - val_loss: 0.3072 - val_acc: 0.9310

Epoch 00046: val_acc did not improve
Epoch 47/100
 - 3s - loss: 0.0057 - acc: 0.9980 - val_loss: 0.3344 - val_acc: 0.9230

Epoch 00047: val_acc did not improve
Epoch 48/100
 - 3s - loss: 0.0109 - acc: 0.9970 - val_loss: 0.3495 - val_acc: 0.9260

Epoch 00048: val_acc did not improve
Epoch 49/100
 - 3s - loss: 0.0055 - acc: 0.9978 - val_loss: 0.3466 - val_acc: 0.9180

Epoch 00049: val_acc did not improve
Epoch 50/100
 - 3s - loss: 0.0169 - acc: 0.9955 - val_loss: 0.3192 - val_acc: 0.9250

Epoch 00050: val_acc did not improve
Epoch 51/100
 - 3s - loss: 0.0080 - acc: 0.9975 - val_loss: 0.3249 - val_acc: 0.9200

Epoch 00051: val_acc did not improve
Epoch 52/100
 - 3s - loss: 0.0122 - acc: 0.9973 - val_loss: 0.2956 - val_acc: 0.9250

Epoch 00052: val_acc did not improve
Epoch 53/100
 - 3s - loss: 0.0153 - acc: 0.9955 - val_loss: 0.2968 - val_acc: 0.9280

Epoch 00053: val_acc did not improve
Epoch 54/100
 - 3s - loss: 0.0073 - acc: 0.9975 - val_loss: 0.3652 - val_acc: 0.9220

Epoch 00054: val_acc did not improve
Epoch 55/100
 - 3s - loss: 0.0143 - acc: 0.9963 - val_loss: 0.3400 - val_acc: 0.9290

Epoch 00055: val_acc did not improve
Epoch 56/100
 - 3s - loss: 0.0055 - acc: 0.9978 - val_loss: 0.3374 - val_acc: 0.9290

Epoch 00056: val_acc did not improve
Epoch 57/100
 - 3s - loss: 0.0086 - acc: 0.9975 - val_loss: 0.3312 - val_acc: 0.9300

Epoch 00057: val_acc did not improve
Epoch 58/100
 - 3s - loss: 0.0100 - acc: 0.9973 - val_loss: 0.4208 - val_acc: 0.9230

Epoch 00058: val_acc did not improve
Epoch 59/100
 - 3s - loss: 0.0093 - acc: 0.9963 - val_loss: 0.3613 - val_acc: 0.9260

Epoch 00059: val_acc did not improve
Epoch 60/100
 - 3s - loss: 0.0117 - acc: 0.9975 - val_loss: 0.3530 - val_acc: 0.9250

Epoch 00060: val_acc did not improve
Epoch 61/100
 - 3s - loss: 0.0086 - acc: 0.9975 - val_loss: 0.3675 - val_acc: 0.9230

Epoch 00061: val_acc did not improve
Epoch 62/100
 - 3s - loss: 0.0215 - acc: 0.9955 - val_loss: 0.3421 - val_acc: 0.9280

Epoch 00062: val_acc did not improve
Epoch 63/100
 - 3s - loss: 0.0165 - acc: 0.9945 - val_loss: 0.3231 - val_acc: 0.9300

Epoch 00063: val_acc did not improve
Epoch 64/100
 - 3s - loss: 0.0091 - acc: 0.9973 - val_loss: 0.3429 - val_acc: 0.9280

Epoch 00064: val_acc did not improve
Epoch 65/100
 - 3s - loss: 0.0120 - acc: 0.9950 - val_loss: 0.3913 - val_acc: 0.9180

Epoch 00065: val_acc did not improve
Epoch 66/100
 - 3s - loss: 0.0133 - acc: 0.9955 - val_loss: 0.3573 - val_acc: 0.9250

Epoch 00066: val_acc did not improve
Epoch 67/100
 - 3s - loss: 0.0107 - acc: 0.9968 - val_loss: 0.3963 - val_acc: 0.9200

Epoch 00067: val_acc did not improve
Epoch 68/100
 - 3s - loss: 0.0072 - acc: 0.9983 - val_loss: 0.3343 - val_acc: 0.9280

Epoch 00068: val_acc did not improve
Epoch 69/100
 - 3s - loss: 0.0080 - acc: 0.9980 - val_loss: 0.3854 - val_acc: 0.9260

Epoch 00069: val_acc did not improve
Epoch 70/100
 - 3s - loss: 0.0097 - acc: 0.9955 - val_loss: 0.3811 - val_acc: 0.9190

Epoch 00070: val_acc did not improve
Epoch 71/100
 - 3s - loss: 0.0099 - acc: 0.9965 - val_loss: 0.3632 - val_acc: 0.9350

Epoch 00071: val_acc did not improve
Epoch 72/100
 - 3s - loss: 0.0083 - acc: 0.9975 - val_loss: 0.3044 - val_acc: 0.9280

Epoch 00072: val_acc did not improve
Epoch 73/100
 - 3s - loss: 0.0057 - acc: 0.9980 - val_loss: 0.3282 - val_acc: 0.9260

Epoch 00073: val_acc did not improve
Epoch 74/100
 - 3s - loss: 0.0083 - acc: 0.9975 - val_loss: 0.2980 - val_acc: 0.9300

Epoch 00074: val_acc did not improve
Epoch 75/100
 - 3s - loss: 0.0057 - acc: 0.9983 - val_loss: 0.3328 - val_acc: 0.9320

Epoch 00075: val_acc did not improve
Epoch 76/100
 - 3s - loss: 0.0065 - acc: 0.9980 - val_loss: 0.2989 - val_acc: 0.9350

Epoch 00076: val_acc did not improve
Epoch 77/100
 - 3s - loss: 0.0074 - acc: 0.9978 - val_loss: 0.4291 - val_acc: 0.9200

Epoch 00077: val_acc did not improve
Epoch 78/100
 - 3s - loss: 0.0037 - acc: 0.9993 - val_loss: 0.3521 - val_acc: 0.9340

Epoch 00078: val_acc did not improve
Epoch 79/100
 - 3s - loss: 0.0103 - acc: 0.9968 - val_loss: 0.3988 - val_acc: 0.9310

Epoch 00079: val_acc did not improve
Epoch 80/100
 - 3s - loss: 0.0152 - acc: 0.9963 - val_loss: 0.3846 - val_acc: 0.9280

Epoch 00080: val_acc did not improve
Epoch 81/100
 - 3s - loss: 0.0078 - acc: 0.9980 - val_loss: 0.4405 - val_acc: 0.9190

Epoch 00081: val_acc did not improve
Epoch 82/100
 - 3s - loss: 0.0127 - acc: 0.9958 - val_loss: 0.3582 - val_acc: 0.9280

Epoch 00082: val_acc did not improve
Epoch 83/100
 - 3s - loss: 0.0065 - acc: 0.9978 - val_loss: 0.3435 - val_acc: 0.9270

Epoch 00083: val_acc did not improve
Epoch 84/100
 - 3s - loss: 0.0098 - acc: 0.9973 - val_loss: 0.3145 - val_acc: 0.9290

Epoch 00084: val_acc did not improve
Epoch 85/100
 - 3s - loss: 0.0079 - acc: 0.9978 - val_loss: 0.3410 - val_acc: 0.9260

Epoch 00085: val_acc did not improve
Epoch 86/100
 - 3s - loss: 0.0087 - acc: 0.9980 - val_loss: 0.3038 - val_acc: 0.9310

Epoch 00086: val_acc did not improve
Epoch 87/100
 - 3s - loss: 0.0072 - acc: 0.9978 - val_loss: 0.3310 - val_acc: 0.9260

Epoch 00087: val_acc did not improve
Epoch 88/100
 - 3s - loss: 0.0046 - acc: 0.9985 - val_loss: 0.3429 - val_acc: 0.9270

Epoch 00088: val_acc did not improve
Epoch 89/100
 - 3s - loss: 0.0049 - acc: 0.9985 - val_loss: 0.4051 - val_acc: 0.9230

Epoch 00089: val_acc did not improve
Epoch 90/100
 - 3s - loss: 0.0136 - acc: 0.9953 - val_loss: 0.3084 - val_acc: 0.9360

Epoch 00090: val_acc improved from 0.93500 to 0.93600, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 91/100
 - 3s - loss: 0.0090 - acc: 0.9975 - val_loss: 0.3203 - val_acc: 0.9310

Epoch 00091: val_acc did not improve
Epoch 92/100
 - 3s - loss: 0.0144 - acc: 0.9963 - val_loss: 0.3448 - val_acc: 0.9270

Epoch 00092: val_acc did not improve
Epoch 93/100
 - 3s - loss: 0.0052 - acc: 0.9983 - val_loss: 0.3274 - val_acc: 0.9310

Epoch 00093: val_acc did not improve
Epoch 94/100
 - 3s - loss: 0.0056 - acc: 0.9988 - val_loss: 0.2924 - val_acc: 0.9400

Epoch 00094: val_acc improved from 0.93600 to 0.94000, saving model to ./stl10/best_model_saved/ResNet50_STL-10_model_best.hdf5
Epoch 95/100
 - 3s - loss: 0.0045 - acc: 0.9983 - val_loss: 0.2844 - val_acc: 0.9370

Epoch 00095: val_acc did not improve
Epoch 96/100
 - 3s - loss: 0.0081 - acc: 0.9968 - val_loss: 0.3102 - val_acc: 0.9270

Epoch 00096: val_acc did not improve
Epoch 97/100
 - 3s - loss: 0.0099 - acc: 0.9963 - val_loss: 0.3719 - val_acc: 0.9250

Epoch 00097: val_acc did not improve
Epoch 98/100
 - 3s - loss: 0.0070 - acc: 0.9990 - val_loss: 0.3383 - val_acc: 0.9220

Epoch 00098: val_acc did not improve
Epoch 99/100
 - 3s - loss: 0.0037 - acc: 0.9990 - val_loss: 0.3100 - val_acc: 0.9340

Epoch 00099: val_acc did not improve
Epoch 100/100
 - 3s - loss: 0.0068 - acc: 0.9980 - val_loss: 0.3542 - val_acc: 0.9230

Epoch 00100: val_acc did not improve
Out[206]:
<keras.callbacks.History at 0x1be00362278>

9.3. Test features

In [41]:
test_features = x.extract_test_features(x_test)
Bottleneck features detected (test)
Bottleneck features (test) loaded

9.3.1. Test finetuned model

In [42]:
# load the weights that yielded the best validation accuracy
# model.load_weights(config['ResNet50']['model_file_saved'])
modelResNet50.load_weights(config['ResNet50']['model_file_saved'])

# evaluate test accuracy
score = modelResNet50.evaluate(test_features, y_test_categorical, verbose=2)
accuracy = 100*score[1]

# print test accuracy
print('Test accuracy: %.4f%%' % accuracy)
Test accuracy: 93.8500%

9.4. Load unlabeled data

In [43]:
x_unlabeled = load_data.read_all_images(config['data_directory'] + '/unlabeled_X.bin')
print('Unlabeled data loaded', x_unlabeled.shape)
Unlabeled data loaded (100000, 96, 96, 3)

9.4.0 Class to process and label unlabeled data

In [12]:
class SummaryUnlabeled:
    def __init__(self):
        self.labels_text = ['Airplane', 'Bird', 'Car', 'Cat', 'Deer', 'Dog', 'Horse', 'Monkey', 'Ship', 'Truck']
        
    def random_data(self, n=10):
        images = []
        for i in range(n):
            index = random.randint(0, len(x_unlabeled))
            image = x_unlabeled[index]
            images.append(image)
        return images
    
    def get_labels(self, images):
        labels = []
        values = []
        for i in range(len(images)):
            pic_features = x.extract_image_features(images[i])
            preds = modelResNet50.predict(np.expand_dims(pic_features,axis=0))
            topclass_idx = np.argmax(preds)
            topclass_prob = np.max(preds)
            labels.append(topclass_idx)
            values.append(topclass_prob)
        return labels, values
    
    def get_indices(self, threshold, how_many):
        indices = []
        count = 0
        for i in range(0,len(x_unlabeled)):
            pic_features = x.extract_image_features(x_unlabeled[i])
            preds = modelResNet50.predict(np.expand_dims(pic_features,axis=0))
            topclass_idx = np.argmax(preds)
            topclass_prob = np.max(preds)
            if topclass_prob >= threshold:
                indices.append(i)
                count = count + 1
                if count==how_many:
                    return indices
        return indices

    def display_data(self, images, labels, values, n_rows, n_cols):
        # show image of 'count' random data points
        plt.figure(figsize=(30, 30), dpi=80)
        fig, axs = plt.subplots(n_rows, n_cols, figsize=(30, 30))
        fig.subplots_adjust(hspace = .25, wspace=.25)
        axs = axs.ravel()

        for i in range(n_cols*n_rows):
            axs[i].axis('off')
            axs[i].imshow(images[i])
            str1 = self.labels_text[labels[i]] + ' (' + str(np.around(values[i], decimals=3))+')'
            axs[i].set_title(str1)
    
    def display_data_more(self, images, labels, values, figh, figw, n_rows, n_cols):
        # show image of 'count' random data points
        plt.figure(figsize=(figw, figh), dpi=128)
        fig, axs = plt.subplots(n_rows, n_cols, figsize=(figw, figh))
        fig.subplots_adjust(hspace = .20, wspace=.15)
        axs = axs.ravel()

        for i in range(n_cols*n_rows):
            axs[i].axis('off')
            axs[i].imshow(images[i])
            str1 = self.labels_text[labels[i]] + ' (' + str(np.around(values[i], decimals=3))+')'
            axs[i].set_title(str1, fontsize=20)

9.4.1. Application of the finetuned model on the Unlebeled data

In [50]:
s = SummaryUnlabeled()
images = s.random_data(200)
labels, values = s.get_labels(images)
s.display_data_more(images, labels, values, 60, 30, 20, 10)
D:\anaconda3\envs\tensorflow-gpu\lib\site-packages\ipykernel_launcher.py:41: DeprecationWarning: `imresize` is deprecated!
`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.
Use ``skimage.transform.resize`` instead.
<Figure size 3840x7680 with 0 Axes>

Analysis:

Given the unlabeled dataset contains objects beyond the 10 object classes available in the labeled STL-10 dataset, it is natural that there are serveral approximations done by the ResNet50-based classifier (which is also only 93% correct against the test subset of STL-10 dataset). Now, the important question is how close is the approximation with respect to the features of the corresponding STL-10 label when the softmax probability for the labeling is close to 1.0, say more than 0.90. We list a few cases (since the images are random, the cases below may not exactly correspond to different runs of the above images):

  • Cat: Mongoose, Cougar, Raccoon, Iguana, Rat, Squirrel, Lioness
  • Horse: Cow, Donkey, Byson,
  • Deer: Goat, Alpaka,
  • Monkey: Chimpanzee, Baboon, Gorilla
  • Bird: Cobra, Iguana

It appears that the softmax probability is higher of features are more and more similar to the corresponding STL-10 class.

9.4.2. Labelling unlabeled images based on softmax threshold

In [48]:
indices = s.get_indices(threshold=0.999, how_many=1000)
D:\anaconda3\envs\tensorflow-gpu\lib\site-packages\ipykernel_launcher.py:41: DeprecationWarning: `imresize` is deprecated!
`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.
Use ``skimage.transform.resize`` instead.
In [49]:
images = []
for i in range(len(indices)):
    images.append(x_unlabeled[indices[i]])
labels, values = s.get_labels(images)
s.display_data_more(images, labels, values, 60, 30, 20, 10)
D:\anaconda3\envs\tensorflow-gpu\lib\site-packages\ipykernel_launcher.py:41: DeprecationWarning: `imresize` is deprecated!
`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.
Use ``skimage.transform.resize`` instead.
<Figure size 3840x7680 with 0 Axes>

Analysis

Since the image distribution is similar but more diverse in the unlabeled STL-10 dataset, we have a number of wrong identifications with very high probability (0.999 or more). Let us look at a few such cases:

  • Cat: Cheetah, Leopard, Raccoon, Lioness, Cougar
  • Deer: Goat, Sheep, Alpaka
  • Dog: Bear, Wolf, Panda
  • Horse: Cow, Donkey, Rhino
  • Monkey: Chimpanzee, Baboon, Gorilla
  • Truck: Train, Jeep, Bus

Even though in the test subset, there is no Cheetah, Leopard, Raccoon, Lioness, Cougar, adding these images as cats in the training dataset may enhance and diversify the features of cats and improve performance on the test subset. A similar assumption may be applicable for the classes: dog, horse, monkey, deer, truck, etc.

10. Augmentation for the model from Scratch

Since we have a large unlabeled image dataset, we restrict our augmentation of the training dataset on adding images selected from that unlabeled dataset based on the following assumptions:

  • Feature-extraction from a pre-trained imagenet model and finetuning it for the STL-10 classes will result
    • A reliable classifier for the unlabeled STL-10 data to label with one of the 10 classes from STL-10 dataset
  • Images classified with high softmax probability (0.999 or more) has one of the following properties:
    • The image belongs to the correct STL-10 class
    • The image shares great similarities in features of the corresponding correct STL-10 class
  • Adding imperfections in the training dataset diversifies relevant features and improves test accuracy
    • Example: a wolf is not a dog, but a certain dog breed may look like a wolf

10.1. Class to handle data augmentation from unlabeled data

In [20]:
class AugmentTraining:
    def __init__(self, threshold, how_many):
        self.threshold = threshold
        self.how_many = how_many
        
    def get_labeled_images(self):
        print("Collecting images")
        s = SummaryUnlabeled()
        indices = s.get_indices(self.threshold, self.how_many)
        images = []
        for i in range(len(indices)):
            images.append(x_unlabeled[indices[i]])
        labels, values = s.get_labels(images)
        print("Images collected")
        return images, labels

    def get_preprocessed_augmented_training_data(self):
        print("Augmenting training data")
        x_train_augmented = x_train.tolist()
        y_train_augmented = y_train.tolist()
        for i in range(len(images)):
            x_train_augmented.append(images[i])
            y_train_augmented.append(labels[i])    
        x_train_augmented = np.array(x_train_augmented)
        y_train_augmented = np.array(y_train_augmented)
        print("Augmentation completed")
        print("Preprocessing augmented data")
        x_train_augmented_normalized = (x_train_augmented)/255.0
        y_train_augmented_categorical = np_utils.to_categorical(y_train_augmented, 10)
        print("Preprocessing completed")
        return x_train_augmented_normalized, y_train_augmented_categorical

10.2. Augment data

In [14]:
def labels_frequency(y):
    # histogram of label frequency
    hist, bins = np.histogram(y, bins=10)
    width = 0.75 * (bins[1] - bins[0])
    center = (bins[:-1] + bins[1:]) / 2
    plt.bar(center, hist, align='center', width=width)
    plt.show()

10.2.1. Add 4000 images with softmax probability at least 0.999

In [17]:
if os.path.isfile(config['augmented_train_data'] + '/train_9000.npz'):
    x_train_augmented_normalized = np.load(config['augmented_train_data'] + '/train_9000.npz')['x'] 
    y_train_augmented_categorical = np.load(config['augmented_train_data'] + '/train_9000.npz')['y']
    print('Loaded: ', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
else:
    aug = AugmentTraining(0.999, 4000)
    images, labels = aug.get_labeled_images()
    x_train_augmented_normalized, y_train_augmented_categorical = aug.get_preprocessed_augmented_training_data()
    images = None
    lables = None
    print('Computed', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
    np.savez_compressed(config['augmented_train_data'] + "/train_9000.npz", 
                        x=x_train_augmented_normalized, 
                        y=y_train_augmented_categorical)
    
labels_frequency(np.argmax(y_train_augmented_categorical, axis=1))
Loaded:  (9000, 96, 96, 3)   (9000, 10)

10.2.1.1 Fit augmented data (9000 x 96 x 96 x 3)

In [25]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_9000.hdf5'):
    pass
else:
    model = ModelFromScratch()
    model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
    model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
    model.fit_augmented('model_80x3_4_1280_relu_augmented_9000.hdf5', 50)
Train on 7200 samples, validate on 1800 samples
Epoch 1/50
 - 18s - loss: 3.5410 - acc: 0.2615 - val_loss: 1.8979 - val_acc: 0.4206

Epoch 00001: val_acc improved from -inf to 0.42056, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 2/50
 - 17s - loss: 2.2568 - acc: 0.3865 - val_loss: 1.5314 - val_acc: 0.4556

Epoch 00002: val_acc improved from 0.42056 to 0.45556, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 3/50
 - 16s - loss: 1.8414 - acc: 0.4299 - val_loss: 1.4336 - val_acc: 0.4939

Epoch 00003: val_acc improved from 0.45556 to 0.49389, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 4/50
 - 17s - loss: 1.5509 - acc: 0.4819 - val_loss: 1.4387 - val_acc: 0.4922

Epoch 00004: val_acc did not improve
Epoch 5/50
 - 17s - loss: 1.3930 - acc: 0.5068 - val_loss: 1.3708 - val_acc: 0.5044

Epoch 00005: val_acc improved from 0.49389 to 0.50444, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 6/50
 - 17s - loss: 1.3018 - acc: 0.5363 - val_loss: 1.3483 - val_acc: 0.5211

Epoch 00006: val_acc improved from 0.50444 to 0.52111, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 7/50
 - 16s - loss: 1.2034 - acc: 0.5679 - val_loss: 1.2983 - val_acc: 0.5294

Epoch 00007: val_acc improved from 0.52111 to 0.52944, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 8/50
 - 16s - loss: 1.1196 - acc: 0.5979 - val_loss: 1.2578 - val_acc: 0.5444

Epoch 00008: val_acc improved from 0.52944 to 0.54444, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 9/50
 - 17s - loss: 1.0605 - acc: 0.6204 - val_loss: 1.2047 - val_acc: 0.5583

Epoch 00009: val_acc improved from 0.54444 to 0.55833, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 10/50
 - 17s - loss: 0.9880 - acc: 0.6435 - val_loss: 1.1794 - val_acc: 0.5678

Epoch 00010: val_acc improved from 0.55833 to 0.56778, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 11/50
 - 17s - loss: 0.9379 - acc: 0.6626 - val_loss: 1.1667 - val_acc: 0.5761

Epoch 00011: val_acc improved from 0.56778 to 0.57611, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 12/50
 - 17s - loss: 0.8871 - acc: 0.6826 - val_loss: 1.1730 - val_acc: 0.5739

Epoch 00012: val_acc did not improve
Epoch 13/50
 - 17s - loss: 0.8379 - acc: 0.6997 - val_loss: 1.1842 - val_acc: 0.5683

Epoch 00013: val_acc did not improve
Epoch 14/50
 - 17s - loss: 0.7885 - acc: 0.7185 - val_loss: 1.1331 - val_acc: 0.5994

Epoch 00014: val_acc improved from 0.57611 to 0.59944, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 15/50
 - 17s - loss: 0.7587 - acc: 0.7194 - val_loss: 1.1486 - val_acc: 0.5939

Epoch 00015: val_acc did not improve
Epoch 16/50
 - 17s - loss: 0.6991 - acc: 0.7468 - val_loss: 1.1040 - val_acc: 0.6056

Epoch 00016: val_acc improved from 0.59944 to 0.60556, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 17/50
 - 16s - loss: 0.6726 - acc: 0.7586 - val_loss: 1.1154 - val_acc: 0.6056

Epoch 00017: val_acc did not improve
Epoch 18/50
 - 16s - loss: 0.6206 - acc: 0.7808 - val_loss: 1.1288 - val_acc: 0.6089

Epoch 00018: val_acc improved from 0.60556 to 0.60889, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 19/50
 - 16s - loss: 0.5833 - acc: 0.7887 - val_loss: 1.1437 - val_acc: 0.6067

Epoch 00019: val_acc did not improve
Epoch 20/50
 - 16s - loss: 0.5642 - acc: 0.7978 - val_loss: 1.1165 - val_acc: 0.6211

Epoch 00020: val_acc improved from 0.60889 to 0.62111, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 21/50
 - 16s - loss: 0.5066 - acc: 0.8218 - val_loss: 1.1143 - val_acc: 0.6144

Epoch 00021: val_acc did not improve
Epoch 22/50
 - 16s - loss: 0.4911 - acc: 0.8251 - val_loss: 1.1045 - val_acc: 0.6011

Epoch 00022: val_acc did not improve
Epoch 23/50
 - 16s - loss: 0.4557 - acc: 0.8390 - val_loss: 1.0933 - val_acc: 0.6217

Epoch 00023: val_acc improved from 0.62111 to 0.62167, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 24/50
 - 17s - loss: 0.4213 - acc: 0.8543 - val_loss: 1.0731 - val_acc: 0.6272

Epoch 00024: val_acc improved from 0.62167 to 0.62722, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 25/50
 - 16s - loss: 0.3806 - acc: 0.8681 - val_loss: 1.0746 - val_acc: 0.6356

Epoch 00025: val_acc improved from 0.62722 to 0.63556, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 26/50
 - 16s - loss: 0.3700 - acc: 0.8715 - val_loss: 1.0698 - val_acc: 0.6361

Epoch 00026: val_acc improved from 0.63556 to 0.63611, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 27/50
 - 17s - loss: 0.3493 - acc: 0.8765 - val_loss: 1.0836 - val_acc: 0.6300

Epoch 00027: val_acc did not improve
Epoch 28/50
 - 17s - loss: 0.3210 - acc: 0.8881 - val_loss: 1.1097 - val_acc: 0.6367

Epoch 00028: val_acc improved from 0.63611 to 0.63667, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 29/50
 - 16s - loss: 0.3014 - acc: 0.8992 - val_loss: 1.0953 - val_acc: 0.6394

Epoch 00029: val_acc improved from 0.63667 to 0.63944, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 30/50
 - 16s - loss: 0.2778 - acc: 0.9075 - val_loss: 1.0821 - val_acc: 0.6400

Epoch 00030: val_acc improved from 0.63944 to 0.64000, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 31/50
 - 16s - loss: 0.2514 - acc: 0.9186 - val_loss: 1.0910 - val_acc: 0.6322

Epoch 00031: val_acc did not improve
Epoch 32/50
 - 16s - loss: 0.2400 - acc: 0.9187 - val_loss: 1.1158 - val_acc: 0.6311

Epoch 00032: val_acc did not improve
Epoch 33/50
 - 16s - loss: 0.2336 - acc: 0.9207 - val_loss: 1.1323 - val_acc: 0.6317

Epoch 00033: val_acc did not improve
Epoch 34/50
 - 16s - loss: 0.2254 - acc: 0.9215 - val_loss: 1.1152 - val_acc: 0.6333

Epoch 00034: val_acc did not improve
Epoch 35/50
 - 17s - loss: 0.1999 - acc: 0.9321 - val_loss: 1.1121 - val_acc: 0.6450

Epoch 00035: val_acc improved from 0.64000 to 0.64500, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 36/50
 - 16s - loss: 0.1961 - acc: 0.9329 - val_loss: 1.1303 - val_acc: 0.6439

Epoch 00036: val_acc did not improve
Epoch 37/50
 - 17s - loss: 0.1747 - acc: 0.9421 - val_loss: 1.1685 - val_acc: 0.6428

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 17s - loss: 0.1703 - acc: 0.9422 - val_loss: 1.1283 - val_acc: 0.6522

Epoch 00038: val_acc improved from 0.64500 to 0.65222, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5
Epoch 39/50
 - 17s - loss: 0.1709 - acc: 0.9418 - val_loss: 1.1601 - val_acc: 0.6461

Epoch 00039: val_acc did not improve
Epoch 40/50
 - 17s - loss: 0.1510 - acc: 0.9508 - val_loss: 1.1750 - val_acc: 0.6433

Epoch 00040: val_acc did not improve
Epoch 41/50
 - 17s - loss: 0.1460 - acc: 0.9510 - val_loss: 1.1864 - val_acc: 0.6378

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 17s - loss: 0.1435 - acc: 0.9496 - val_loss: 1.1775 - val_acc: 0.6439

Epoch 00042: val_acc did not improve
Epoch 43/50
 - 17s - loss: 0.1328 - acc: 0.9568 - val_loss: 1.2454 - val_acc: 0.6367

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 17s - loss: 0.1249 - acc: 0.9611 - val_loss: 1.2226 - val_acc: 0.6289

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 17s - loss: 0.1206 - acc: 0.9608 - val_loss: 1.1840 - val_acc: 0.6422

Epoch 00045: val_acc did not improve
Epoch 46/50
 - 17s - loss: 0.1080 - acc: 0.9668 - val_loss: 1.2095 - val_acc: 0.6361

Epoch 00046: val_acc did not improve
Epoch 47/50
 - 17s - loss: 0.1145 - acc: 0.9613 - val_loss: 1.2126 - val_acc: 0.6450

Epoch 00047: val_acc did not improve
Epoch 48/50
 - 17s - loss: 0.1035 - acc: 0.9656 - val_loss: 1.2273 - val_acc: 0.6489

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 17s - loss: 0.1040 - acc: 0.9663 - val_loss: 1.2601 - val_acc: 0.6400

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 17s - loss: 0.0939 - acc: 0.9685 - val_loss: 1.1868 - val_acc: 0.6567

Epoch 00050: val_acc improved from 0.65222 to 0.65667, saving model to model_80x3_4_1280_relu_augmented_9000.hdf5

10.2.1.2. Test using weights trained from augmented data (9000 x 96 x96 x3)

In [56]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_9000.hdf5'):
    model.test('model_80x3_4_1280_relu_augmented_9000.hdf5')
else:
    pass
Test accuracy: 67.2000%

10.2.2. Add 8000 images with softmax probability at least 0.999

In [18]:
if os.path.isfile(config['augmented_train_data'] + '/train_13000.npz'):
    x_train_augmented_normalized = np.load(config['augmented_train_data'] + '/train_13000.npz')['x'] 
    y_train_augmented_categorical = np.load(config['augmented_train_data'] + '/train_13000.npz')['y']
    print('Loaded: ', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
else:
    aug = AugmentTraining(0.999, 8000)
    images, labels = aug.get_labeled_images()
    x_train_augmented_normalized, y_train_augmented_categorical = aug.get_preprocessed_augmented_training_data()
    images = None
    lables = None
    print('Computed', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
    np.savez_compressed(config['augmented_train_data'] + "/train_13000.npz", 
                        x=x_train_augmented_normalized, 
                        y=y_train_augmented_categorical)

labels_frequency(np.argmax(y_train_augmented_categorical, axis=1))
Loaded:  (13000, 96, 96, 3)   (13000, 10)

10.2.2.1 Fit augmented data (13000 x 96 x 96 x 3)

In [28]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_13000.hdf5'):
    pass
else:
    model = ModelFromScratch()
    model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
    model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
    model.fit_augmented('model_80x3_4_1280_relu_augmented_13000.hdf5', 50)
Train on 10400 samples, validate on 2600 samples
Epoch 1/50
 - 25s - loss: 3.2591 - acc: 0.2868 - val_loss: 1.6643 - val_acc: 0.4381

Epoch 00001: val_acc improved from -inf to 0.43808, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 2/50
 - 24s - loss: 1.9792 - acc: 0.4078 - val_loss: 1.4157 - val_acc: 0.4900

Epoch 00002: val_acc improved from 0.43808 to 0.49000, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 3/50
 - 24s - loss: 1.6191 - acc: 0.4472 - val_loss: 1.4169 - val_acc: 0.5169

Epoch 00003: val_acc improved from 0.49000 to 0.51692, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 4/50
 - 24s - loss: 1.4218 - acc: 0.4960 - val_loss: 1.3196 - val_acc: 0.5300

Epoch 00004: val_acc improved from 0.51692 to 0.53000, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 5/50
 - 24s - loss: 1.3150 - acc: 0.5267 - val_loss: 1.2419 - val_acc: 0.5592

Epoch 00005: val_acc improved from 0.53000 to 0.55923, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 6/50
 - 25s - loss: 1.2214 - acc: 0.5573 - val_loss: 1.2353 - val_acc: 0.5731

Epoch 00006: val_acc improved from 0.55923 to 0.57308, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 7/50
 - 24s - loss: 1.1378 - acc: 0.5899 - val_loss: 1.1511 - val_acc: 0.5823

Epoch 00007: val_acc improved from 0.57308 to 0.58231, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 8/50
 - 24s - loss: 1.0725 - acc: 0.6152 - val_loss: 1.0938 - val_acc: 0.6154

Epoch 00008: val_acc improved from 0.58231 to 0.61538, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 9/50
 - 24s - loss: 1.0156 - acc: 0.6309 - val_loss: 1.0707 - val_acc: 0.6154

Epoch 00009: val_acc improved from 0.61538 to 0.61538, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 10/50
 - 24s - loss: 0.9505 - acc: 0.6585 - val_loss: 1.0432 - val_acc: 0.6273

Epoch 00010: val_acc improved from 0.61538 to 0.62731, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 11/50
 - 24s - loss: 0.8958 - acc: 0.6783 - val_loss: 1.0318 - val_acc: 0.6308

Epoch 00011: val_acc improved from 0.62731 to 0.63077, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 12/50
 - 24s - loss: 0.8431 - acc: 0.6991 - val_loss: 1.0126 - val_acc: 0.6473

Epoch 00012: val_acc improved from 0.63077 to 0.64731, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 13/50
 - 24s - loss: 0.7962 - acc: 0.7113 - val_loss: 0.9795 - val_acc: 0.6523

Epoch 00013: val_acc improved from 0.64731 to 0.65231, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 14/50
 - 24s - loss: 0.7647 - acc: 0.7285 - val_loss: 1.0208 - val_acc: 0.6381

Epoch 00014: val_acc did not improve
Epoch 15/50
 - 24s - loss: 0.7129 - acc: 0.7474 - val_loss: 0.9789 - val_acc: 0.6523

Epoch 00015: val_acc did not improve
Epoch 16/50
 - 24s - loss: 0.6834 - acc: 0.7558 - val_loss: 0.9857 - val_acc: 0.6623

Epoch 00016: val_acc improved from 0.65231 to 0.66231, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 17/50
 - 24s - loss: 0.6329 - acc: 0.7744 - val_loss: 0.9728 - val_acc: 0.6581

Epoch 00017: val_acc did not improve
Epoch 18/50
 - 24s - loss: 0.6090 - acc: 0.7808 - val_loss: 0.9223 - val_acc: 0.6750

Epoch 00018: val_acc improved from 0.66231 to 0.67500, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 19/50
 - 24s - loss: 0.5598 - acc: 0.8012 - val_loss: 0.9795 - val_acc: 0.6585

Epoch 00019: val_acc did not improve
Epoch 20/50
 - 24s - loss: 0.5273 - acc: 0.8109 - val_loss: 0.9330 - val_acc: 0.6704

Epoch 00020: val_acc did not improve
Epoch 21/50
 - 24s - loss: 0.4801 - acc: 0.8317 - val_loss: 0.8965 - val_acc: 0.6808

Epoch 00021: val_acc improved from 0.67500 to 0.68077, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 22/50
 - 24s - loss: 0.4485 - acc: 0.8375 - val_loss: 0.9175 - val_acc: 0.6731

Epoch 00022: val_acc did not improve
Epoch 23/50
 - 24s - loss: 0.4312 - acc: 0.8476 - val_loss: 0.9630 - val_acc: 0.6665

Epoch 00023: val_acc did not improve
Epoch 24/50
 - 24s - loss: 0.3975 - acc: 0.8653 - val_loss: 0.8823 - val_acc: 0.6958

Epoch 00024: val_acc improved from 0.68077 to 0.69577, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 25/50
 - 24s - loss: 0.3703 - acc: 0.8697 - val_loss: 0.9762 - val_acc: 0.6685

Epoch 00025: val_acc did not improve
Epoch 26/50
 - 24s - loss: 0.3595 - acc: 0.8740 - val_loss: 0.9193 - val_acc: 0.6804

Epoch 00026: val_acc did not improve
Epoch 27/50
 - 24s - loss: 0.3293 - acc: 0.8841 - val_loss: 0.9061 - val_acc: 0.6969

Epoch 00027: val_acc improved from 0.69577 to 0.69692, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 28/50
 - 24s - loss: 0.3075 - acc: 0.8912 - val_loss: 0.9531 - val_acc: 0.6785

Epoch 00028: val_acc did not improve
Epoch 29/50
 - 24s - loss: 0.2894 - acc: 0.8991 - val_loss: 0.9279 - val_acc: 0.6954

Epoch 00029: val_acc did not improve
Epoch 30/50
 - 24s - loss: 0.2727 - acc: 0.9032 - val_loss: 0.9477 - val_acc: 0.6854

Epoch 00030: val_acc did not improve
Epoch 31/50
 - 24s - loss: 0.2484 - acc: 0.9149 - val_loss: 0.9132 - val_acc: 0.6981

Epoch 00031: val_acc improved from 0.69692 to 0.69808, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 32/50
 - 24s - loss: 0.2410 - acc: 0.9157 - val_loss: 0.9740 - val_acc: 0.6823

Epoch 00032: val_acc did not improve
Epoch 33/50
 - 24s - loss: 0.2245 - acc: 0.9217 - val_loss: 0.9069 - val_acc: 0.6992

Epoch 00033: val_acc improved from 0.69808 to 0.69923, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 34/50
 - 24s - loss: 0.2049 - acc: 0.9284 - val_loss: 0.9542 - val_acc: 0.6896

Epoch 00034: val_acc did not improve
Epoch 35/50
 - 24s - loss: 0.1916 - acc: 0.9371 - val_loss: 0.9445 - val_acc: 0.6908

Epoch 00035: val_acc did not improve
Epoch 36/50
 - 24s - loss: 0.1975 - acc: 0.9330 - val_loss: 0.9318 - val_acc: 0.7027

Epoch 00036: val_acc improved from 0.69923 to 0.70269, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 37/50
 - 24s - loss: 0.1767 - acc: 0.9416 - val_loss: 0.9375 - val_acc: 0.6977

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 24s - loss: 0.1611 - acc: 0.9448 - val_loss: 0.9425 - val_acc: 0.7023

Epoch 00038: val_acc did not improve
Epoch 39/50
 - 24s - loss: 0.1537 - acc: 0.9484 - val_loss: 0.9621 - val_acc: 0.6946

Epoch 00039: val_acc did not improve
Epoch 40/50
 - 24s - loss: 0.1540 - acc: 0.9491 - val_loss: 0.9678 - val_acc: 0.6950

Epoch 00040: val_acc did not improve
Epoch 41/50
 - 24s - loss: 0.1438 - acc: 0.9515 - val_loss: 0.9522 - val_acc: 0.6935

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 24s - loss: 0.1437 - acc: 0.9503 - val_loss: 0.9843 - val_acc: 0.6919

Epoch 00042: val_acc did not improve
Epoch 43/50
 - 24s - loss: 0.1276 - acc: 0.9568 - val_loss: 1.0465 - val_acc: 0.6885

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 24s - loss: 0.1307 - acc: 0.9564 - val_loss: 0.9622 - val_acc: 0.6988

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 24s - loss: 0.1165 - acc: 0.9615 - val_loss: 0.9556 - val_acc: 0.7042

Epoch 00045: val_acc improved from 0.70269 to 0.70423, saving model to model_80x3_4_1280_relu_augmented_13000.hdf5
Epoch 46/50
 - 24s - loss: 0.1146 - acc: 0.9634 - val_loss: 0.9953 - val_acc: 0.6908

Epoch 00046: val_acc did not improve
Epoch 47/50
 - 24s - loss: 0.1061 - acc: 0.9653 - val_loss: 1.0687 - val_acc: 0.6785

Epoch 00047: val_acc did not improve
Epoch 48/50
 - 24s - loss: 0.1048 - acc: 0.9659 - val_loss: 1.0352 - val_acc: 0.6946

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 24s - loss: 0.1011 - acc: 0.9675 - val_loss: 1.0834 - val_acc: 0.6850

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 24s - loss: 0.0992 - acc: 0.9654 - val_loss: 1.0065 - val_acc: 0.7027

Epoch 00050: val_acc did not improve

10.2.2.2. Test using weights trained from augmented data (13000 x 96 x96 x3)

In [29]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_13000.hdf5'):
    model.test('model_80x3_4_1280_relu_augmented_13000.hdf5')
else:
    pass
Test accuracy: 68.9250%

10.2.3. Add 12000 images with softmax probability at least 0.999

In [19]:
if os.path.isfile(config['augmented_train_data'] + '/train_17000.npz'):
    x_train_augmented_normalized = np.load(config['augmented_train_data'] + '/train_17000.npz')['x'] 
    y_train_augmented_categorical = np.load(config['augmented_train_data'] + '/train_17000.npz')['y']
    print('Loaded: ', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
else:
    aug = AugmentTraining(0.999, 12000)
    images, labels = aug.get_labeled_images()
    x_train_augmented_normalized, y_train_augmented_categorical = aug.get_preprocessed_augmented_training_data()
    images = None
    lables = None
    print('Computed', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
    np.savez_compressed(config['augmented_train_data'] + "/train_17000.npz", 
                        x=x_train_augmented_normalized, 
                        y=y_train_augmented_categorical)

labels_frequency(np.argmax(y_train_augmented_categorical, axis=1))
Loaded:  (17000, 96, 96, 3)   (17000, 10)

10.2.3.1 Fit augmented data (17000 x 96 x 96 x 3)

In [22]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_17000.hdf5'):
    pass
else:
    model = ModelFromScratch()
    model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
    model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
    model.fit_augmented('model_80x3_4_1280_relu_augmented_17000.hdf5', 50)
Train on 13600 samples, validate on 3400 samples
Epoch 1/50
 - 38s - loss: 3.1239 - acc: 0.3078 - val_loss: 1.7024 - val_acc: 0.4194

Epoch 00001: val_acc improved from -inf to 0.41941, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 2/50
 - 31s - loss: 1.8781 - acc: 0.4043 - val_loss: 1.4923 - val_acc: 0.4721

Epoch 00002: val_acc improved from 0.41941 to 0.47206, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 3/50
 - 32s - loss: 1.5358 - acc: 0.4588 - val_loss: 1.4228 - val_acc: 0.4829

Epoch 00003: val_acc improved from 0.47206 to 0.48294, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 4/50
 - 32s - loss: 1.3811 - acc: 0.5029 - val_loss: 1.3184 - val_acc: 0.5353

Epoch 00004: val_acc improved from 0.48294 to 0.53529, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 5/50
 - 32s - loss: 1.2710 - acc: 0.5384 - val_loss: 1.2320 - val_acc: 0.5644

Epoch 00005: val_acc improved from 0.53529 to 0.56441, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 6/50
 - 31s - loss: 1.1931 - acc: 0.5748 - val_loss: 1.1757 - val_acc: 0.5835

Epoch 00006: val_acc improved from 0.56441 to 0.58353, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 7/50
 - 32s - loss: 1.1080 - acc: 0.6001 - val_loss: 1.1522 - val_acc: 0.5921

Epoch 00007: val_acc improved from 0.58353 to 0.59206, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 8/50
 - 32s - loss: 1.0595 - acc: 0.6194 - val_loss: 1.1211 - val_acc: 0.6035

Epoch 00008: val_acc improved from 0.59206 to 0.60353, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 9/50
 - 32s - loss: 0.9948 - acc: 0.6439 - val_loss: 1.0787 - val_acc: 0.6097

Epoch 00009: val_acc improved from 0.60353 to 0.60971, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 10/50
 - 32s - loss: 0.9359 - acc: 0.6641 - val_loss: 1.0601 - val_acc: 0.6194

Epoch 00010: val_acc improved from 0.60971 to 0.61941, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 11/50
 - 31s - loss: 0.8791 - acc: 0.6837 - val_loss: 1.0113 - val_acc: 0.6365

Epoch 00011: val_acc improved from 0.61941 to 0.63647, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 12/50
 - 32s - loss: 0.8192 - acc: 0.7041 - val_loss: 1.0030 - val_acc: 0.6479

Epoch 00012: val_acc improved from 0.63647 to 0.64794, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 13/50
 - 32s - loss: 0.7821 - acc: 0.7204 - val_loss: 0.9894 - val_acc: 0.6547

Epoch 00013: val_acc improved from 0.64794 to 0.65471, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 14/50
 - 32s - loss: 0.7380 - acc: 0.7378 - val_loss: 0.9946 - val_acc: 0.6444

Epoch 00014: val_acc did not improve
Epoch 15/50
 - 32s - loss: 0.6906 - acc: 0.7526 - val_loss: 0.9386 - val_acc: 0.6659

Epoch 00015: val_acc improved from 0.65471 to 0.66588, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 16/50
 - 32s - loss: 0.6404 - acc: 0.7722 - val_loss: 0.9501 - val_acc: 0.6635

Epoch 00016: val_acc did not improve
Epoch 17/50
 - 32s - loss: 0.6214 - acc: 0.7791 - val_loss: 0.9408 - val_acc: 0.6671

Epoch 00017: val_acc improved from 0.66588 to 0.66706, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 18/50
 - 32s - loss: 0.5869 - acc: 0.7896 - val_loss: 1.0089 - val_acc: 0.6512

Epoch 00018: val_acc did not improve
Epoch 19/50
 - 32s - loss: 0.5356 - acc: 0.8102 - val_loss: 0.8920 - val_acc: 0.6921

Epoch 00019: val_acc improved from 0.66706 to 0.69206, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 20/50
 - 32s - loss: 0.5122 - acc: 0.8186 - val_loss: 0.9161 - val_acc: 0.6826

Epoch 00020: val_acc did not improve
Epoch 21/50
 - 32s - loss: 0.4618 - acc: 0.8371 - val_loss: 0.8857 - val_acc: 0.6956

Epoch 00021: val_acc improved from 0.69206 to 0.69559, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 22/50
 - 31s - loss: 0.4403 - acc: 0.8454 - val_loss: 0.8816 - val_acc: 0.6965

Epoch 00022: val_acc improved from 0.69559 to 0.69647, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 23/50
 - 31s - loss: 0.4010 - acc: 0.8582 - val_loss: 0.8990 - val_acc: 0.6900

Epoch 00023: val_acc did not improve
Epoch 24/50
 - 31s - loss: 0.3748 - acc: 0.8669 - val_loss: 0.9125 - val_acc: 0.6915

Epoch 00024: val_acc did not improve
Epoch 25/50
 - 31s - loss: 0.3514 - acc: 0.8727 - val_loss: 0.9174 - val_acc: 0.6894

Epoch 00025: val_acc did not improve
Epoch 26/50
 - 32s - loss: 0.3382 - acc: 0.8811 - val_loss: 0.9532 - val_acc: 0.6768

Epoch 00026: val_acc did not improve
Epoch 27/50
 - 32s - loss: 0.3123 - acc: 0.8906 - val_loss: 0.8982 - val_acc: 0.6968

Epoch 00027: val_acc improved from 0.69647 to 0.69676, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 28/50
 - 32s - loss: 0.2908 - acc: 0.8975 - val_loss: 0.8906 - val_acc: 0.7003

Epoch 00028: val_acc improved from 0.69676 to 0.70029, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 29/50
 - 32s - loss: 0.2710 - acc: 0.9033 - val_loss: 0.9201 - val_acc: 0.7012

Epoch 00029: val_acc improved from 0.70029 to 0.70118, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 30/50
 - 32s - loss: 0.2468 - acc: 0.9128 - val_loss: 0.9046 - val_acc: 0.7024

Epoch 00030: val_acc improved from 0.70118 to 0.70235, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 31/50
 - 32s - loss: 0.2415 - acc: 0.9152 - val_loss: 0.9025 - val_acc: 0.7044

Epoch 00031: val_acc improved from 0.70235 to 0.70441, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 32/50
 - 31s - loss: 0.2262 - acc: 0.9207 - val_loss: 0.9058 - val_acc: 0.7068

Epoch 00032: val_acc improved from 0.70441 to 0.70676, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 33/50
 - 32s - loss: 0.2028 - acc: 0.9304 - val_loss: 0.9550 - val_acc: 0.7000

Epoch 00033: val_acc did not improve
Epoch 34/50
 - 31s - loss: 0.1969 - acc: 0.9332 - val_loss: 0.9010 - val_acc: 0.7115

Epoch 00034: val_acc improved from 0.70676 to 0.71147, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 35/50
 - 32s - loss: 0.1787 - acc: 0.9404 - val_loss: 0.9439 - val_acc: 0.7024

Epoch 00035: val_acc did not improve
Epoch 36/50
 - 32s - loss: 0.1699 - acc: 0.9413 - val_loss: 0.9266 - val_acc: 0.7100

Epoch 00036: val_acc did not improve
Epoch 37/50
 - 32s - loss: 0.1518 - acc: 0.9476 - val_loss: 0.9041 - val_acc: 0.7079

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 32s - loss: 0.1478 - acc: 0.9501 - val_loss: 0.9485 - val_acc: 0.7056

Epoch 00038: val_acc did not improve
Epoch 39/50
 - 32s - loss: 0.1403 - acc: 0.9537 - val_loss: 0.9076 - val_acc: 0.7124

Epoch 00039: val_acc improved from 0.71147 to 0.71235, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 40/50
 - 32s - loss: 0.1402 - acc: 0.9512 - val_loss: 0.9083 - val_acc: 0.7141

Epoch 00040: val_acc improved from 0.71235 to 0.71412, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 41/50
 - 32s - loss: 0.1334 - acc: 0.9557 - val_loss: 0.9324 - val_acc: 0.7115

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 32s - loss: 0.1177 - acc: 0.9618 - val_loss: 1.0869 - val_acc: 0.6844

Epoch 00042: val_acc did not improve
Epoch 43/50
 - 31s - loss: 0.1213 - acc: 0.9607 - val_loss: 0.9490 - val_acc: 0.7106

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 31s - loss: 0.1134 - acc: 0.9632 - val_loss: 0.9892 - val_acc: 0.7126

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 31s - loss: 0.1052 - acc: 0.9659 - val_loss: 0.9476 - val_acc: 0.7100

Epoch 00045: val_acc did not improve
Epoch 46/50
 - 31s - loss: 0.0967 - acc: 0.9671 - val_loss: 0.9218 - val_acc: 0.7144

Epoch 00046: val_acc improved from 0.71412 to 0.71441, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 47/50
 - 32s - loss: 0.0928 - acc: 0.9704 - val_loss: 0.9507 - val_acc: 0.7203

Epoch 00047: val_acc improved from 0.71441 to 0.72029, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5
Epoch 48/50
 - 31s - loss: 0.0921 - acc: 0.9704 - val_loss: 1.0285 - val_acc: 0.7094

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 31s - loss: 0.0951 - acc: 0.9674 - val_loss: 0.9832 - val_acc: 0.7174

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 31s - loss: 0.0880 - acc: 0.9707 - val_loss: 0.9687 - val_acc: 0.7206

Epoch 00050: val_acc improved from 0.72029 to 0.72059, saving model to model_80x3_4_1280_relu_augmented_17000.hdf5

10.2.3.2. Test using weights trained from augmented data (17000 x 96 x96 x3)

In [23]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_17000.hdf5'):
    model.test('model_80x3_4_1280_relu_augmented_17000.hdf5')
else:
    pass
Test accuracy: 71.6375%

10.2.4. Add 16000 images with softmax probability at least 0.999

In [20]:
if os.path.isfile(config['augmented_train_data'] + '/train_21000.npz'):
    x_train_augmented_normalized = np.load(config['augmented_train_data'] + '/train_21000.npz')['x'] 
    y_train_augmented_categorical = np.load(config['augmented_train_data'] + '/train_21000.npz')['y']
    print('Loaded: ', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
else:
    aug = AugmentTraining(0.999, 16000)
    images, labels = aug.get_labeled_images()
    x_train_augmented_normalized, y_train_augmented_categorical = aug.get_preprocessed_augmented_training_data()
    images = None
    lables = None
    print('Computed', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
    np.savez_compressed(config['augmented_train_data'] + "/train_21000.npz", 
                        x=x_train_augmented_normalized, 
                        y=y_train_augmented_categorical)

labels_frequency(np.argmax(y_train_augmented_categorical, axis=1))
Loaded:  (21000, 96, 96, 3)   (21000, 10)

10.2.4.1 Fit augmented data (21000 x 96 x 96 x 3)

In [25]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_21000.hdf5'):
    pass
else:
    model = ModelFromScratch()
    model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
    model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
    model.fit_augmented('model_80x3_4_1280_relu_augmented_21000.hdf5', 50)
Train on 16800 samples, validate on 4200 samples
Epoch 1/50
 - 41s - loss: 2.9664 - acc: 0.3195 - val_loss: 1.4681 - val_acc: 0.4771

Epoch 00001: val_acc improved from -inf to 0.47714, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 2/50
 - 39s - loss: 1.7781 - acc: 0.4367 - val_loss: 1.3708 - val_acc: 0.5174

Epoch 00002: val_acc improved from 0.47714 to 0.51738, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 3/50
 - 39s - loss: 1.4321 - acc: 0.4910 - val_loss: 1.2161 - val_acc: 0.5760

Epoch 00003: val_acc improved from 0.51738 to 0.57595, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 4/50
 - 39s - loss: 1.2908 - acc: 0.5321 - val_loss: 1.1656 - val_acc: 0.5886

Epoch 00004: val_acc improved from 0.57595 to 0.58857, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 5/50
 - 39s - loss: 1.1935 - acc: 0.5669 - val_loss: 1.1018 - val_acc: 0.6160

Epoch 00005: val_acc improved from 0.58857 to 0.61595, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 6/50
 - 39s - loss: 1.1070 - acc: 0.6013 - val_loss: 1.0619 - val_acc: 0.6202

Epoch 00006: val_acc improved from 0.61595 to 0.62024, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 7/50
 - 39s - loss: 1.0273 - acc: 0.6317 - val_loss: 1.0350 - val_acc: 0.6276

Epoch 00007: val_acc improved from 0.62024 to 0.62762, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 8/50
 - 39s - loss: 0.9748 - acc: 0.6523 - val_loss: 0.9955 - val_acc: 0.6400

Epoch 00008: val_acc improved from 0.62762 to 0.64000, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 9/50
 - 39s - loss: 0.9027 - acc: 0.6763 - val_loss: 0.9782 - val_acc: 0.6476

Epoch 00009: val_acc improved from 0.64000 to 0.64762, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 10/50
 - 39s - loss: 0.8482 - acc: 0.6978 - val_loss: 0.9304 - val_acc: 0.6700

Epoch 00010: val_acc improved from 0.64762 to 0.67000, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 11/50
 - 39s - loss: 0.7890 - acc: 0.7149 - val_loss: 0.9231 - val_acc: 0.6690

Epoch 00011: val_acc did not improve
Epoch 12/50
 - 39s - loss: 0.7361 - acc: 0.7357 - val_loss: 0.8745 - val_acc: 0.6850

Epoch 00012: val_acc improved from 0.67000 to 0.68500, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 13/50
 - 39s - loss: 0.6998 - acc: 0.7471 - val_loss: 0.8821 - val_acc: 0.6876

Epoch 00013: val_acc improved from 0.68500 to 0.68762, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 14/50
 - 39s - loss: 0.6638 - acc: 0.7599 - val_loss: 0.8583 - val_acc: 0.6905

Epoch 00014: val_acc improved from 0.68762 to 0.69048, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 15/50
 - 39s - loss: 0.6093 - acc: 0.7848 - val_loss: 0.8625 - val_acc: 0.6862

Epoch 00015: val_acc did not improve
Epoch 16/50
 - 39s - loss: 0.5766 - acc: 0.7920 - val_loss: 0.8686 - val_acc: 0.6971

Epoch 00016: val_acc improved from 0.69048 to 0.69714, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 17/50
 - 39s - loss: 0.5496 - acc: 0.8029 - val_loss: 0.8263 - val_acc: 0.7057

Epoch 00017: val_acc improved from 0.69714 to 0.70571, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 18/50
 - 39s - loss: 0.5046 - acc: 0.8211 - val_loss: 0.8540 - val_acc: 0.6998

Epoch 00018: val_acc did not improve
Epoch 19/50
 - 39s - loss: 0.4662 - acc: 0.8328 - val_loss: 0.8140 - val_acc: 0.7179

Epoch 00019: val_acc improved from 0.70571 to 0.71786, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 20/50
 - 39s - loss: 0.4402 - acc: 0.8414 - val_loss: 0.8341 - val_acc: 0.7060

Epoch 00020: val_acc did not improve
Epoch 21/50
 - 39s - loss: 0.4116 - acc: 0.8551 - val_loss: 0.8091 - val_acc: 0.7129

Epoch 00021: val_acc did not improve
Epoch 22/50
 - 40s - loss: 0.3807 - acc: 0.8621 - val_loss: 0.8674 - val_acc: 0.7055

Epoch 00022: val_acc did not improve
Epoch 23/50
 - 40s - loss: 0.3579 - acc: 0.8743 - val_loss: 0.8172 - val_acc: 0.7214

Epoch 00023: val_acc improved from 0.71786 to 0.72143, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 24/50
 - 39s - loss: 0.3245 - acc: 0.8849 - val_loss: 0.8132 - val_acc: 0.7200

Epoch 00024: val_acc did not improve
Epoch 25/50
 - 39s - loss: 0.3112 - acc: 0.8925 - val_loss: 0.9038 - val_acc: 0.6979

Epoch 00025: val_acc did not improve
Epoch 26/50
 - 39s - loss: 0.3016 - acc: 0.8934 - val_loss: 0.8378 - val_acc: 0.7152

Epoch 00026: val_acc did not improve
Epoch 27/50
 - 39s - loss: 0.2759 - acc: 0.9040 - val_loss: 0.8745 - val_acc: 0.7143

Epoch 00027: val_acc did not improve
Epoch 28/50
 - 39s - loss: 0.2540 - acc: 0.9101 - val_loss: 0.8366 - val_acc: 0.7255

Epoch 00028: val_acc improved from 0.72143 to 0.72548, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 29/50
 - 39s - loss: 0.2469 - acc: 0.9144 - val_loss: 0.8442 - val_acc: 0.7281

Epoch 00029: val_acc improved from 0.72548 to 0.72810, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 30/50
 - 40s - loss: 0.2188 - acc: 0.9228 - val_loss: 0.8364 - val_acc: 0.7233

Epoch 00030: val_acc did not improve
Epoch 31/50
 - 40s - loss: 0.2060 - acc: 0.9289 - val_loss: 0.8799 - val_acc: 0.7219

Epoch 00031: val_acc did not improve
Epoch 32/50
 - 39s - loss: 0.1947 - acc: 0.9321 - val_loss: 0.8618 - val_acc: 0.7198

Epoch 00032: val_acc did not improve
Epoch 33/50
 - 39s - loss: 0.1807 - acc: 0.9388 - val_loss: 0.8436 - val_acc: 0.7295

Epoch 00033: val_acc improved from 0.72810 to 0.72952, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 34/50
 - 39s - loss: 0.1704 - acc: 0.9414 - val_loss: 0.8599 - val_acc: 0.7362

Epoch 00034: val_acc improved from 0.72952 to 0.73619, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 35/50
 - 39s - loss: 0.1584 - acc: 0.9463 - val_loss: 0.8735 - val_acc: 0.7360

Epoch 00035: val_acc did not improve
Epoch 36/50
 - 39s - loss: 0.1491 - acc: 0.9509 - val_loss: 0.8308 - val_acc: 0.7367

Epoch 00036: val_acc improved from 0.73619 to 0.73667, saving model to model_80x3_4_1280_relu_augmented_21000.hdf5
Epoch 37/50
 - 39s - loss: 0.1451 - acc: 0.9494 - val_loss: 0.8461 - val_acc: 0.7324

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 39s - loss: 0.1293 - acc: 0.9560 - val_loss: 0.9773 - val_acc: 0.7240

Epoch 00038: val_acc did not improve
Epoch 39/50
 - 39s - loss: 0.1309 - acc: 0.9547 - val_loss: 0.8556 - val_acc: 0.7290

Epoch 00039: val_acc did not improve
Epoch 40/50
 - 39s - loss: 0.1228 - acc: 0.9596 - val_loss: 0.9132 - val_acc: 0.7238

Epoch 00040: val_acc did not improve
Epoch 41/50
 - 39s - loss: 0.1219 - acc: 0.9595 - val_loss: 0.9136 - val_acc: 0.7286

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 40s - loss: 0.1181 - acc: 0.9604 - val_loss: 0.8967 - val_acc: 0.7245

Epoch 00042: val_acc did not improve
Epoch 43/50
 - 39s - loss: 0.1159 - acc: 0.9612 - val_loss: 0.8717 - val_acc: 0.7319

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 39s - loss: 0.1057 - acc: 0.9654 - val_loss: 0.9059 - val_acc: 0.7336

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 39s - loss: 0.1023 - acc: 0.9665 - val_loss: 0.9280 - val_acc: 0.7300

Epoch 00045: val_acc did not improve
Epoch 46/50
 - 39s - loss: 0.0904 - acc: 0.9705 - val_loss: 0.9721 - val_acc: 0.7314

Epoch 00046: val_acc did not improve
Epoch 47/50
 - 40s - loss: 0.0945 - acc: 0.9671 - val_loss: 0.9473 - val_acc: 0.7317

Epoch 00047: val_acc did not improve
Epoch 48/50
 - 39s - loss: 0.0909 - acc: 0.9707 - val_loss: 0.9939 - val_acc: 0.7179

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 39s - loss: 0.0844 - acc: 0.9719 - val_loss: 1.0178 - val_acc: 0.7129

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 39s - loss: 0.0850 - acc: 0.9724 - val_loss: 0.9447 - val_acc: 0.7248

Epoch 00050: val_acc did not improve

10.2.4.2. Test using weights trained from augmented data (21000 x 96 x96 x3)

In [26]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_21000.hdf5'):
    model.test('model_80x3_4_1280_relu_augmented_21000.hdf5')
else:
    pass
Test accuracy: 73.2375%

10.2.5. Add 20000 images with softmax probability at least 0.999

In [21]:
if os.path.isfile(config['augmented_train_data'] + '/train_25000.npz'):
    x_train_augmented_normalized = np.load(config['augmented_train_data'] + '/train_25000.npz')['x'] 
    y_train_augmented_categorical = np.load(config['augmented_train_data'] + '/train_25000.npz')['y']
    print('Loaded: ', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
else:
    aug = AugmentTraining(0.999, 20000)
    images, labels = aug.get_labeled_images()
    x_train_augmented_normalized, y_train_augmented_categorical = aug.get_preprocessed_augmented_training_data()
    images = None
    lables = None
    print('Computed', x_train_augmented_normalized.shape, ' ', y_train_augmented_categorical.shape)
    np.savez_compressed(config['augmented_train_data'] + "/train_25000.npz", 
                        x=x_train_augmented_normalized, 
                        y=y_train_augmented_categorical)

labels_frequency(np.argmax(y_train_augmented_categorical, axis=1))
Loaded:  (25000, 96, 96, 3)   (25000, 10)

10.2.5.1 Fit augmented data (25000 x 96 x 96 x 3)

In [23]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_25000.hdf5'):
    pass
else:
    model = ModelFromScratch()
    model.build(x_train_normalized.shape[1:], [80, 80, 80], 4, [1280], 'relu', 10)
    model.compile(rate=0.00009, beta_1=0.99, beta_2=0.999)
    model.fit_augmented('model_80x3_4_1280_relu_augmented_25000.hdf5', 50)
Train on 20000 samples, validate on 5000 samples
Epoch 1/50
 - 50s - loss: 2.8322 - acc: 0.3238 - val_loss: 1.5380 - val_acc: 0.4384

Epoch 00001: val_acc improved from -inf to 0.43840, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 2/50
 - 46s - loss: 1.6594 - acc: 0.4380 - val_loss: 1.3995 - val_acc: 0.5160

Epoch 00002: val_acc improved from 0.43840 to 0.51600, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 3/50
 - 46s - loss: 1.3955 - acc: 0.4999 - val_loss: 1.2645 - val_acc: 0.5574

Epoch 00003: val_acc improved from 0.51600 to 0.55740, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 4/50
 - 46s - loss: 1.2603 - acc: 0.5467 - val_loss: 1.2014 - val_acc: 0.5744

Epoch 00004: val_acc improved from 0.55740 to 0.57440, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 5/50
 - 46s - loss: 1.1538 - acc: 0.5843 - val_loss: 1.1436 - val_acc: 0.5978

Epoch 00005: val_acc improved from 0.57440 to 0.59780, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 6/50
 - 46s - loss: 1.0650 - acc: 0.6158 - val_loss: 1.0331 - val_acc: 0.6358

Epoch 00006: val_acc improved from 0.59780 to 0.63580, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 7/50
 - 46s - loss: 0.9931 - acc: 0.6456 - val_loss: 1.0017 - val_acc: 0.6378

Epoch 00007: val_acc improved from 0.63580 to 0.63780, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 8/50
 - 46s - loss: 0.9350 - acc: 0.6631 - val_loss: 1.0180 - val_acc: 0.6368

Epoch 00008: val_acc did not improve
Epoch 9/50
 - 46s - loss: 0.8828 - acc: 0.6847 - val_loss: 0.9571 - val_acc: 0.6618

Epoch 00009: val_acc improved from 0.63780 to 0.66180, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 10/50
 - 46s - loss: 0.8216 - acc: 0.7046 - val_loss: 0.9623 - val_acc: 0.6598

Epoch 00010: val_acc did not improve
Epoch 11/50
 - 46s - loss: 0.7735 - acc: 0.7217 - val_loss: 0.9052 - val_acc: 0.6746

Epoch 00011: val_acc improved from 0.66180 to 0.67460, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 12/50
 - 46s - loss: 0.7308 - acc: 0.7391 - val_loss: 0.9242 - val_acc: 0.6710

Epoch 00012: val_acc did not improve
Epoch 13/50
 - 46s - loss: 0.6788 - acc: 0.7547 - val_loss: 0.8664 - val_acc: 0.6892

Epoch 00013: val_acc improved from 0.67460 to 0.68920, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 14/50
 - 46s - loss: 0.6385 - acc: 0.7719 - val_loss: 0.8644 - val_acc: 0.6888

Epoch 00014: val_acc did not improve
Epoch 15/50
 - 46s - loss: 0.5897 - acc: 0.7880 - val_loss: 0.8145 - val_acc: 0.7118

Epoch 00015: val_acc improved from 0.68920 to 0.71180, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 16/50
 - 46s - loss: 0.5581 - acc: 0.7983 - val_loss: 0.8210 - val_acc: 0.7106

Epoch 00016: val_acc did not improve
Epoch 17/50
 - 47s - loss: 0.5257 - acc: 0.8119 - val_loss: 0.8350 - val_acc: 0.7040

Epoch 00017: val_acc did not improve
Epoch 18/50
 - 47s - loss: 0.4947 - acc: 0.8219 - val_loss: 0.8091 - val_acc: 0.7138

Epoch 00018: val_acc improved from 0.71180 to 0.71380, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 19/50
 - 46s - loss: 0.4504 - acc: 0.8397 - val_loss: 0.8241 - val_acc: 0.7198

Epoch 00019: val_acc improved from 0.71380 to 0.71980, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 20/50
 - 46s - loss: 0.4214 - acc: 0.8479 - val_loss: 0.8025 - val_acc: 0.7194

Epoch 00020: val_acc did not improve
Epoch 21/50
 - 46s - loss: 0.3988 - acc: 0.8564 - val_loss: 0.8983 - val_acc: 0.7036

Epoch 00021: val_acc did not improve
Epoch 22/50
 - 46s - loss: 0.3685 - acc: 0.8685 - val_loss: 0.8529 - val_acc: 0.7102

Epoch 00022: val_acc did not improve
Epoch 23/50
 - 46s - loss: 0.3310 - acc: 0.8826 - val_loss: 0.8713 - val_acc: 0.7114

Epoch 00023: val_acc did not improve
Epoch 24/50
 - 46s - loss: 0.3147 - acc: 0.8859 - val_loss: 0.8870 - val_acc: 0.7078

Epoch 00024: val_acc did not improve
Epoch 25/50
 - 46s - loss: 0.2929 - acc: 0.8971 - val_loss: 0.8566 - val_acc: 0.7168

Epoch 00025: val_acc did not improve
Epoch 26/50
 - 46s - loss: 0.2780 - acc: 0.9034 - val_loss: 0.8138 - val_acc: 0.7276

Epoch 00026: val_acc improved from 0.71980 to 0.72760, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 27/50
 - 46s - loss: 0.2536 - acc: 0.9123 - val_loss: 0.7862 - val_acc: 0.7336

Epoch 00027: val_acc improved from 0.72760 to 0.73360, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 28/50
 - 46s - loss: 0.2397 - acc: 0.9153 - val_loss: 0.8077 - val_acc: 0.7364

Epoch 00028: val_acc improved from 0.73360 to 0.73640, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 29/50
 - 46s - loss: 0.2157 - acc: 0.9242 - val_loss: 0.8069 - val_acc: 0.7312

Epoch 00029: val_acc did not improve
Epoch 30/50
 - 46s - loss: 0.2013 - acc: 0.9346 - val_loss: 0.8883 - val_acc: 0.7156

Epoch 00030: val_acc did not improve
Epoch 31/50
 - 46s - loss: 0.1884 - acc: 0.9352 - val_loss: 0.8293 - val_acc: 0.7378

Epoch 00031: val_acc improved from 0.73640 to 0.73780, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 32/50
 - 46s - loss: 0.1857 - acc: 0.9348 - val_loss: 0.8387 - val_acc: 0.7356

Epoch 00032: val_acc did not improve
Epoch 33/50
 - 46s - loss: 0.1729 - acc: 0.9402 - val_loss: 0.8134 - val_acc: 0.7406

Epoch 00033: val_acc improved from 0.73780 to 0.74060, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 34/50
 - 46s - loss: 0.1585 - acc: 0.9461 - val_loss: 0.8549 - val_acc: 0.7360

Epoch 00034: val_acc did not improve
Epoch 35/50
 - 46s - loss: 0.1485 - acc: 0.9517 - val_loss: 0.8264 - val_acc: 0.7308

Epoch 00035: val_acc did not improve
Epoch 36/50
 - 46s - loss: 0.1437 - acc: 0.9506 - val_loss: 0.8677 - val_acc: 0.7402

Epoch 00036: val_acc did not improve
Epoch 37/50
 - 46s - loss: 0.1346 - acc: 0.9560 - val_loss: 0.9213 - val_acc: 0.7278

Epoch 00037: val_acc did not improve
Epoch 38/50
 - 46s - loss: 0.1393 - acc: 0.9525 - val_loss: 0.8848 - val_acc: 0.7298

Epoch 00038: val_acc did not improve
Epoch 39/50
 - 46s - loss: 0.1229 - acc: 0.9584 - val_loss: 0.8931 - val_acc: 0.7354

Epoch 00039: val_acc did not improve
Epoch 40/50
 - 46s - loss: 0.1202 - acc: 0.9588 - val_loss: 0.8926 - val_acc: 0.7372

Epoch 00040: val_acc did not improve
Epoch 41/50
 - 46s - loss: 0.1174 - acc: 0.9609 - val_loss: 1.0402 - val_acc: 0.7134

Epoch 00041: val_acc did not improve
Epoch 42/50
 - 46s - loss: 0.1078 - acc: 0.9642 - val_loss: 0.8724 - val_acc: 0.7310

Epoch 00042: val_acc did not improve
Epoch 43/50
 - 46s - loss: 0.0997 - acc: 0.9675 - val_loss: 0.9026 - val_acc: 0.7328

Epoch 00043: val_acc did not improve
Epoch 44/50
 - 46s - loss: 0.0947 - acc: 0.9686 - val_loss: 0.8680 - val_acc: 0.7384

Epoch 00044: val_acc did not improve
Epoch 45/50
 - 46s - loss: 0.0948 - acc: 0.9683 - val_loss: 0.9272 - val_acc: 0.7380

Epoch 00045: val_acc did not improve
Epoch 46/50
 - 46s - loss: 0.0887 - acc: 0.9695 - val_loss: 0.9341 - val_acc: 0.7424

Epoch 00046: val_acc improved from 0.74060 to 0.74240, saving model to model_80x3_4_1280_relu_augmented_25000.hdf5
Epoch 47/50
 - 46s - loss: 0.0837 - acc: 0.9723 - val_loss: 0.9433 - val_acc: 0.7320

Epoch 00047: val_acc did not improve
Epoch 48/50
 - 46s - loss: 0.0892 - acc: 0.9698 - val_loss: 1.0886 - val_acc: 0.7168

Epoch 00048: val_acc did not improve
Epoch 49/50
 - 46s - loss: 0.0842 - acc: 0.9717 - val_loss: 0.9728 - val_acc: 0.7350

Epoch 00049: val_acc did not improve
Epoch 50/50
 - 46s - loss: 0.0805 - acc: 0.9733 - val_loss: 0.9710 - val_acc: 0.7382

Epoch 00050: val_acc did not improve

10.2.5.2. Test using weights trained from augmented data (25000 x 96 x96 x3)

In [13]:
if os.path.isfile('model_80x3_4_1280_relu_augmented_25000.hdf5'):
    model.test('model_80x3_4_1280_relu_augmented_25000.hdf5')
else:
    pass
Test accuracy: 75.6375%

11. Performance summary of the CNN Model from Scratch

Training set-size Epochs Training loss Training accuracy Validation loss Validation accuracy Test accuracy
5000 (original) 50 0.1259 96.10% 1.1916 63.300% 61.600%
9000 50 0.0939 96.85% 1.1868 65.667% 67.200%
13000 50 0.1165 96.15% 0.9556 70.425% 68.925%
17000 50 0.0880 97.07% 0.9687 72.059% 71.638%
21000 50 0.1491 95.09% 0.8308 73.667% 73.238%
25000 50 0.0887 96.95% 0.9341 74.200% 75.638%

Observations

  • The model obtains a test accuracy more than 60% on the original training subset
  • Validation accuracy on the original as well as the augmented training datasets generalize really good on the training subset
  • Test accuracy increases as more relevant images from the unlabeled dataset is added to the training subset
  • Adding imperfections in the training dataset does not make any negative impact on the model from scratch
    • As anticipated, imperfections have actually helped to diversify features for the dataset classes
      • Example: Identifying a bear or a wolf as a dog
      • Example: Identifying a donkey or a cow as a horse
      • Example: Identifying a cheetah or a leopard as a cat

11.1. Some pictures from the web

In [63]:
images = s.images_from_a_folder('./images/*.*')
labels = []
for i in range(len(images)):
    img = np.expand_dims(images[i], axis=0)
    img_class = np.argmax(model.predict(img))
    labels.append(img_class)
    
s.display_data(images, labels, 1, len(images))
<Figure size 3840x1920 with 0 Axes>

With only 75% accuracy, it is expected that on randomly selected images, there will be misclassifications. Yet, the model seems to perform quite well as in the set of internet pictures (resized to 96 x 96 x 3)

12. Conclusion

Due to limitations in time and computing resources, we could not perform further experiments, but we have achieved the goals of this project and found a constructive way to improve performance of a model from scratch to perform consistantly better on the STL-10 dataset. In other words, we have determined a scalable method to obtain better performance on the dataset.

Note that we have a state-of-the-art performance using a model from scratch on the STL-10 dataset but only with help of transfer learning while understanding the unlabeled data. We have obtained this performance without applying popular preprocessing and image augmentations techniques.

Since we have devised a process involving a basic model from scratch, we may consider the following to improve performance further:

  • How we initialize the weights, learning rates, and other parameters? (say, we restart fitting with the previous best weight)
  • Tune the structure of the Symmetric model from scratch
  • Given required resources are available, popular preprocessing and augmentation techniques can be used